F/G  15/7 


A087  886  PEACEPTRONICS  INC  400DLAND  HILLS  CALIF 

EMPIRICAL  EVALUATION  OF  A  DECISION-ANALYTIC  AID.(U) 
may  80  F  G  CHRISTEN*  M  G  SAMET  NO 00 14-78— C-0 380 

UNCLASSIFIED  PFTR-1 066-80-1  NL 


H 


nil 


IE 


ADA  0  8  78  8  6 


Final  Technical  Report  PFTR- 1 068-80- 1 
Contract  No.  N00014-78-C-0340 
Project  NO.  NR  197-048 
May.  1980 


EMPIRICAL  EVALUATION  OF 
A  DECISION-ANALYTIC  AID 


Francois  Q.  Christen 
Michael  G.  Samet 


S  « 


PREPARED  FOR: 

OFFICE  OF  NAVAL  RESEARCH,  CODE  455 
800  North  Quincy  Street  Room  711 
Artngton,  VA  22217 


PERCEPTRONICS 


T 


TO  ."If  I  "'.'1  TIT 


TTTTT 


Final  Technical  Report  PFTR- 1066-80-1 
Contract  No.  N00014-78-C-0340 
Project  No.  NR  197-048 
May,  1980 


EMPIRICAL  EVALUATION  OF 
A  DECISION-ANALYTIC  AID 


Francois  G.  Christen 
Michael  G.  Samet 


PREPARED  FOR: 


OFFICE  OF  NAVAL  RESEARCH,  CODE  465 
800  North  Quincy  Street  Room  711 
Arlington,  VA  22217 


PERCEPTRONICS 


6271  VARIEL  AVENUE  •  WOODLAND  HILLS  •  CALIFORNIA  91367  •  PHONE  (213)  884-7470 


Unclassified 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  fWhtn  D.l*  E nl.f.d) _ 

REPORT  DOCUMENTATION  PAGE _ before0 completSgNform 

A._JS£*©*X- NUMBER- - /  |2.  GOVT  ACCESSION  NO.  S.  RE CIPIENT'S  CATALOG  NUMBER 


L_  JtERORT  ■  NUMBER-  : 


PFTR- 1066-80-1 


4.  UTILB  (fid  Sufi  tit)  I 


5.  J:yPEJ3F.  BEPSJRT  A  PERIOD  COVERED 


Empirical  Evaluation  of  a  Decision-Analytic  Aid  #  ^Technical  ,Repi?t#j 


">■  author/*; 

>  Francois  G.  /Christen 
/•  (Michael  G./Samet 


».  PERFORMING  ORGANIZATION  NAME  ANO  ADDRESS 

Perceptronics,  Inc." 

6271  Variel  Avenue 

Woodland  Hills,  California  91367 

II.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

Office  of  Naval  Research,  Code  455., 


S.  CONT  RAG-TOR -GRANT-NUMBERS  I 

yi'j  N/0^14-78-C-/034O  J 


10.  PROGRAM  ELEMENT.  PROJECT,  TASK 
AREA  A  WORK  UNIT  NUMBERS 


'll)  A  .  M  NR  197-048 


12.  REPORT  DATE 

May,  1980 


800  No.  Quincy  Street,  Room  711  )'  )  )  .  //  "number  of  'pages - 

Arlington,  Virginia  22217  •"  103 

monitoring  agency  name  A  AOORESS/lf  dllltrtnt  Item  Controlling  OtOct)  IS.  SECURITY  CLASS,  (ot  Ihit  rtport) 

Unclassified 


I  16.  DISTRIBUTION  STATEMENT  (of  thit  Report) 


ISA.  OECLASSI  FI  CATION.  DOWNGRADING 

scneoule 


Approved  for  Public  Release.  Distribution  unlimited. 


IT.  DISTRIBUTION  STATEMENT  (of  the  ebettmct  entered  In  Block  20,  II  different  from  Roport) 


ir  surplementarv  notes 


I  IS.  KEY  WOROS  f  Confirm#  on  rower*  e  ilrfa  If  neeaiiary  mnd  Identify  by  block  number) 


Alternative  Selection 
Computer-Aided  Decisions 
Decision  Aid 
Decision  Analysis 


Decision  Evaluation 
Decision  Making 
Decision  Model 
Decision  Scenario 


Interactive  Computers 
Subjective  Probability 
Utility  Estimation 


20  ABSTRACT  f  Confirm#  on  rower  to  wide  if  neceeemry  and  Identity  by  bloek  number) 

An  experiment  was  performed  to  Investigate  the  potential  of  DDI's  Rapid 
Screening  of  Options  (0PINT)  minicomputer-based  aiding  package  for  enhancing 
decision  making.  Two  different  politico-military  scenario  backgrounds  and 
message  sets  were  developed  (familiar  and  unfamiliar),  and  each  was  further 
elaborated  and  configured  into  a  set  of  five  intelligence  surmaries  (i.e., 
scenario  version),  which  cumulatively  reflected  an  enemy  intention  of  either 
"attack"  or  "no  attack."  The  ground  truth  for  each  scenario  version  was  - 


00  ,  1473  EOITION  OF  I  NOV  SS  IS  OBSOLETE 


Unclassified 


Unclassified 


security  classification  of  this  nag er*t>*<  o««  Ent»nd) 
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criterion  for  evaluating  performance  in  the  subsequent  main  experiment.  The 
subjects  in  the  main  experiment  were  24  experienced  naval  intelligence  analysts ,{ 
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control  condition.  Each  analyst  was  given  one  version  of  the  familiar  scenario 
and  the  opposite  version  of  the  unfamiliar  scenario.  After  reading  each 
intelligence  summary,  he  was  required  to  diagnose  the  enemy's  military  inten¬ 
tions  and  to  make  a  decision  by  recommending  one  of  four  prespecified  courses 
of  action.  The  use  of  the  aiding  package  significantly  increased  the  number 
of  correct  decisions  under  the  attack  version  of  the  scenarios,  but  not  under 
the  no-attack  version;  this  result  was  attributed  to  content  differences 
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of  the  analysts.!  Furthermore,  the  aid  recommended  33%  more  correct  decisions 
than  were  selected  by  the  aided  analysts,  a  finding  which  highlights  the  need 
for  building  useri  confidence  in  an  analytical  aid.  Suggestions  for  future 
research  are  enumerated  including  the  need  to  differentiate  between  a  decision 
aid's  potential  for  supporting  problem  structuring  versus  information  aggre¬ 
gation. 
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ABSTRACT 


An  experiment  was  performed  to  investigate  the  potential  of  DDI's  Rapid 
Screening  of  Options  (OPINT)  minicomputer-based  aiding  package  for 
enhancing  decision  making.  Two  different  politico-military  scenario 
backgrounds  and  message  sets  were  developed  (familiar  and  unfamiliar), 
and  each  was  further  elaborated  and  configured  into  a  set  of  five  intel¬ 
ligence  summaries  (i.e.,  scenario  version),  which  cumulatively  reflected 
an  enemy  intention  of  either  "attack"  or  "no  attack."  The  ground  truth 
for  each  scenario  version  was  then  validated  in  a  preliminary  empirical 
study  so  that  it  could  serve  as  a  criterion  for  evaluating  performance 
in  the  subsequent  main  experiment.  The  subjects  in  the  main  experiment 
were  24  experienced  naval  intelligence  analysts  with  12  participating  in 
an  aided  experimental  condition  and  12  in  an  unaided  control  condition. 
Each  analyst  was  given  one  version  of  the  familiar  scenario  and  the  op¬ 
posite  version  of  the  unfamiliar  scenario.  After  reading  each  intelli¬ 
gence  summary,  he  was  required  to  diagnose  the  enemy's  military  inten¬ 
tions  and  to  make  a  decision  by  recommending  one  of  four  prespecified 
courses  of  action.  The  use  of  the  aiding  package  significantly  in¬ 
creased  the  number  of  correct  decisions  under  the  attack  version  of  the 
scenarios,  but  not  under  the  no-attack  version;  this  result  was  attri¬ 
buted  to  content  differences  between  the  scenarios  as  well  as  to  a  con¬ 
servative  utility  function  on  the  part  of  the  analysts.  Furthermore, 
the  aid  recommended  33%  more  correct  decisions  than  were  selected  by  the 
aided  analysts,  a  finding  which  highlights  the  need  for  building  user 
confidence  in  an  analytical  aid.  Suggestions  for  future  research  are 
enumerated  including  the  need  to  differentiate  between  a  decision  aid's 
potential  for  supporting  problem  structuring  versus  information  aggrega¬ 
tion. 
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INTRODUCTION 


To  perform  adequately,  a  military  commander  must  continually  seek  and 
process  information.  Furthermore,  he  must  often  make  subjective  assess¬ 
ments  such  as  value  judgments  and  consider  trade-offs  which  weigh  heavi¬ 
ly  in  his  decision  making.  One  common  trade-off,  for  example,  relates 
to  risk-benefit  analysis  in  which  the  military  practitioner  balances  po¬ 
tential  benefits  against  risks  when  choosing  courses  of  action.  Whether 
it's  a  strategic  commander  in  a  war-planning  conference  or  a  tactical 
commander  in  the  field,  the  cognitive  processes  which  come  into  play  in 
each  case  are  probably  quite  similar.  In  making  judgments,  commanders 
are  forced  to  rely  for  the  most  part  upon  their  intuitive  capabilities 
as  information  processors  and  decision  makers. 

Because  of  the  implications  of  failures  of  decision  makers  to  analyze 
and  interpret  information  appropriately ,  especially  in  the  military  are¬ 
na,  investigators  have  explored  the  role  that  the  behavioral  study  of 
decision  processes  can  have  in  improving  such  decision  making  (e.g., 
Slovic,  Fischhoff,  and  Lichtenstein,  1977).  Findings  in  the  psychologi¬ 
cal  and  human  factors  literature  suggest  that  people  are  sensitive  to 
key  informational  and  situational  parameters  when  making  decisions. 
However,  they  consistently  show  systematic,  stereotypical  biases  when 
analyzing  available  information  for  decision  making.  Apparently,  be¬ 
cause  of  limited  memory,  attention,  reasoning,  and  computational  capa¬ 
bilities,  humans  are  unable  to  integrate  and  combine  various  dimensions 
of  information  to  arrive  at  a  subjective  assessment  of  the  situation 
that  is  consistent  and  valid.  Therefore,  it  appears  that  humans  require 
normative  support,  wherever  possible,  to  enhance  the  quality  of  their 
information  processing  and  decision  making.  One  way  of  obtaining  such 
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support  is  through  the  medium  of  "decision  aids." 

A  generally  acceptable  definition  of  a  decision  aid  is  difficult  to 
identify.  After  a  comprehensive  search  through  the  literature  on  deci¬ 
sion  aids,  Levit,  Alden,  Erickson  and  Heaton  (1974)  concluded  that 
"...the  working  definition  of  decision  aiding  is  dependent  on  the  as¬ 
sumptions  of  the  decision  making  framework  from  which  it  is  derived." 
However,  most  person-machine  decision  aids  are  conceived  with  the  same 
basic  intention  in  mind:  namely,  to  allocate  information  processing  and 
decision  functions  between  the  human  and  the  machine  in  a  way  that  op¬ 
timizes  the  use  of  their  respective  strengths  and  compensates  for  their 
respective  weaknesses.  The  goal  of  such  a  procedure-oriented  configura¬ 
tion  is  to  maximize  the  joint  performance  of  the  person-machine  system. 

In  its  broadest  sense,  therefore,  a  decision  aid  can  be  defined  as  any 
technique  or  procedure  that  restructures  the  methods  by  which  problems 
are  analyzed,  alternatives  developed,  and  decisions  taken  (Spector, 
Hayes,  and  Crain,  1976).  However,  with  regard  to  military  decision  mak¬ 
ing,  it  is  clear  that  no  single  decision  aiding  concept  would  be  ade¬ 
quate  for  implementation  into  a  command  information  system.  In  fact, 
after  their  review  of  relevant  research,  Levit  et  al  (1974)  concluded: 

"A  complex  of  decision  aids,  integrated  into  a  decision  support  system, 
is  necessary  to  optimize  the  tactical  performance  of  the  human/computer 
decision  making  unit."  This  report  focuses  on  the  evaluation  of  one 
class  of  promising  decision  aids  that  might  contribute  to  such  a  support 
system.  These  aids,  referred  to  as  decision-analytic  aids,  have  been 
designed  from  decision- theoretic  principles,  which  provide  a  framework 
for  mathematical  representation  of  the  decision  making  process. 

Considerable  effort  has  been  devoted  to  the  development  of  decision- 
analytic  aids,  partly  because  their  application  is  expected  to  have  si g- 
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nificant  practical  value  for  military  decisionmaking  (Peterson,  Phil¬ 
lips,  Randall,  and  Shawcross,  1977),  which  is  usually  characterized  by 
"fairly  well-defined  objectives,  significant  action  alternatives,  rela¬ 
tively  high  statures,  inconclusive  information,  and  limited  time  for  de¬ 
cision"  (Schrenk,  1969).  Although  the  problems  that  can  be  addressed  by 
applied  decision  theory  are  complex  and  vexing  (i.e.,  full  of  value 
judgments,  uncertainty,  and  conflict),  the  methodology  employed  is  con¬ 
ceptually  simple  and  not  difficult  to  comprehend  (in  contrast,  for  exam¬ 
ple,  to  artificial  intelligence  techniques).  Perhaps  it  is  this  element 
of  simplicity  that  makes  decision  theoretic  procedures  so  attractive. 

In  essence,  decision  analysis  is  a  technology  which  merges  the  two 
fields  of  decision  theory  and  systems  analysis  (Brown,  Kahr,  and  Peter¬ 
son,  1974;  Keeney  and  Raiffa,  1976).  It  assists  individuals  or  groups 
of  individuals  to  make  up  their  minds,  by  explicating  and  quantifying 
the  considerations,  however  subjective,  that  enter  into  any  decision. 
Decision  analysis  accommodates  the  same  types  of  information,  judgments, 
and  preferences  as  does  informal  decision  making;  however,  it  imposes 
rational  analysis  and  discipline  on  the  reasoning.  Two  major  steps  are 
usually  necessary  in  the  application  of  decision  analysis:  The  first 
involves  structuring  the  problem  by  specifying  decision  alternatives, 
event  outcomes,  and  their  attributes;  the  second  involves  quantifica¬ 
tion,  or  the  assignment  of  a  numerical  scale  for  the  probabilities  and 
utilities  of  event  outcomes.  These  values  can  then  be  aggregated  ac¬ 
cording  to  normative  techniques  (e.g.,  expectation  theory)  to  establish 
recommended  decision  alternatives. 

Thus,  the  objective  of  decision  analysis  is  to  simplify  the  representa¬ 
tion  and  integration  of  what  are  often  complex  configurations  and  in¬ 
terne!  ■■tions  among  relevant  data.  In  short,  "decision  analysis  is  a 
logical  procedure  for  the  balancing  of  factors  which  influence  a  deci- 
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sion"  (Howard,  1968).  Beyond  its  potential  value  in  helping  individual 
decision  makers  to  structure  their  own  information  and  thinking,  deci¬ 
sion  analysis  also  provides  a  vehicle  for  communication  between  indivi¬ 
duals  especially  with  regard  to  the  reasoning  that  underlies  a  decision. 
This  latter  advantage  has  been  referred  to  as  the  "indirect  value"  of 
decision  analysis  (Brown  and  Watson,  1976). 

Because  subjective  probability  and  utility  estimation  are  critical  as¬ 
pects  of  the  decision-analytic  process,  the  potential  of  probability- 
based  and  utility-based  decision  aiding  systems  has  been  seriously  in¬ 
vestigated.  Much  early  work  on  decision  aiding  has  centered  around  Pro¬ 
babilistic  Information  Processing  (PIP)  systems  for  Bayesian  diagnosis 
(Edwards,  1964;  Kaplan  and  Newnan,  1966).  These  systems  involve  the  use 
of  Bayes'  theorem  to  help  the  decision  maker  optimally  revise  opinions 
in  light  of  new  information.  The  concept  requires  that  the  person  esti¬ 
mate  conditional  probabilities  that  specific,  predictive  (diagnostic) 
information  will  be  observed  when  certain  environmental  states  occur. 
These  conditional  probabilities  are  then  transmitted  to  the  computer, 
which  utiltizes  Bayes'  theorem  to  aggregate  the  likelihood  estimates 
with  prior  probabilities  in  order  to  define  posteriority  judgments. 
Following  this  procedure,  it  is  possible  to  compensate  for  human  inabil¬ 
ity  to  retain  and  combine  separate  data  points  into  an  overall  conclu¬ 
sion. 

Several  PIP-type  systems  have  been  demonstrated  for  assisting  people  in 
making  mil itary-rel ated  diagnostic  decisions.  Howell  (1967),  for  exam¬ 
ple,  says  that  improvements  of  around  10-15%  in  diagnostic  decisions  can 
be  expected  with  automated  aggregation.  Improvements  become  particular¬ 
ly  noticeable  under  conditions  of  time  or  load  stress  or  low  input 
fidelity.  Kelly  and  Peterson  (1971)  have  reported  similar  findings  with 
an  intelligence  analysis  system.  Analysts  were  trained  in  the  PIP  tech- 
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nique  and  subsequently  were  tested  in  analysis  of  realistic  intelligence 
problems.  Results  indicated  that  the  PIP  procedure  can  increase  the  ef¬ 
ficiency  with  which  probabilities  or  odds  are  revised  in  light  of  new 
information.  This  type  of  technology  has  been  further  advanced  by  John¬ 
son  and  Hal  pin  (1972)  who  developed  a  multi-state  computer-aided  Baye¬ 
sian  inference  system  for  tactical  intelligence  environments.  Overall, 
the  consensus  in  the  literature  appears  to  favor  the  use  of  probability- 
based  aiding  techniques  to  overcome  the  conservative  bias  often  dis¬ 
played  by  decision  makers  confronted  with  highly  diagnostic  information, 
and  such  systems  have  been  deemed  "potentially  profitable"  in  the  con¬ 
text  of  military  decision  making  (Beach,  1975). 

Assessments  of  risks  and  gains  must  also  be  performed  within  the  frame¬ 
work  of  decision  analysis.  A  logical  companion  for  probability-based 
aiding  thus  involves  the  idea  of  compensating  for  human  deficiencies  in 
utility  assessment.  One  of  the  earliest  utility-based  aids  was 
developed  by  Miller,  Kaplan  and  Edwards  (1967,  1968).  Using  this  sys¬ 
tem,  the  Judged  Utility  Decision  Generate  (JUDGE),  they  successfully 
demonstrated  the  advantage  of  automating  aircraft  dispatch  in  tactical 
air  command  systems.  In  this  situation,  the  commander  was  required  to 
consider  the  relative  value  of  the  targets,  their  probability  of  des¬ 
truction  and  the  number  of  available  aircraft.  In  forming  a  decision 
policy,  the  key  human  inputs  were  the  utility  of  destruction  of  various 
targets.  Based  on  the  value  judgments  of  trained  personnel  as  well  as 
other  inputs,  the  JUDGE  model  selected  a  course  of  action  which  maxim¬ 
ized  the  expected  utility.  The  underlying  rationale  is  that  the  person 
observes  the  real  world,  estimates  its  states  and  their  associated  pro¬ 
babilities  and  utilities,  and  provides  them  as  inputs  to  a  computer  al¬ 
gorithm  which  then  generates  the  decision  output  (Schum,  1970).  Experi¬ 
ments  with  experienced  tactical  air  controllers  revealed  that  JUDGE  did 
a  more  effective  job  than  did  people  working  without  it. 


Recently,  considerable  research  emphasis  has  shifted  to  the  theoretical 
and  applied  aspects  of  multi-attribute  utility  (MAU)  assessment  tech¬ 
niques  (e.g.,  Johnson  and  Huber,  1977;  Keeney  and  Raiffa,  1977).  This 
technology  has  been  offered  as  a  potentially  better  way  to  assess  utili¬ 
ty  than  the  intuitive- based  “global"  and  "holistic"  assessment  method 
that  is  commonly  used.  The  intent  is  to  facilitate  a  complex  subjective 
evaluation  by  implementing  a  "divide  and  conquer"  approach  to  human 
judgment.  Typically,  the  procedure  involves  the  decomposition  of  each 
decision  outcome  into  a  set  of  dimensions  or  attributes  that  can  be 
separately  assessed--both  in  terms  of  their  importance  (i.e.,  weight)  in 
a  given  situation,  and  their  quantitative  level  with  respect  to  the  out¬ 
come.  The  attribute  weightings  and  corresponding  levels  are  measured 
and  then  combined  by  an  appropriate  assessment  model  (e.g.,  a  linear  ad¬ 
ditive  rule)  to  provide  a  single  utility  for  the  multi-attribute  out¬ 
come.  The  assessment  model  serves  as  a  normative  aid  which  aggregates 
the  human  judgments. 

There  is  evidence  to  suggest  that  MAU  procedures,  when  compared  with  the 
"holistic"  or  "global"  approach,  may  enhance  the  dependability  or  con¬ 
vergent  validity  of  utility  estimates  (Fischer,  1979;  Newman,  1975; 
Samet,  1976).  The  theoretical  rationale  is  that  decomposition  of  the 
decision  alternatives  into  their  component  attributes  enables  the  deci¬ 
sion  maker  to  assign  utilities  in  a  relatively  unambiguous  fashion.  The 
result  appears  to  be  increased  consistency  of  intra-  and  well  as  inter¬ 
subject  judgments.  Hence,  utility-based  decision  aids  may  offer  a  com¬ 
mander  an  alternative  to  using  intuitive  military  judgment  by  providing 
a  systematic,  analytical  procedure  by  which  the  pros  and  cons  of  a  pros¬ 
pective  course  of  action  can  be  weighted  in  order  to  differentiate 
between  a  good  strategy  or  tactic  and  a  bad  one.  For  this  reason,  the 
MAU  approach  for  assessing  subjective  judgment,  has  been  applied  in  a 
variety  of  military  domains  (e.g.,  Barclay,  Brown,  Kelly,  Peterson, 
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Phillips,  and  Selvidge,  1977;  Brown,  Kelly,  Stewart,  and  Uvila,  1975; 
Hays,  O'Connor,  and  Peterson,  1975;  O'Connor  and  Buede,  1977). 


Evaluative  Issues 


The  measure  of  human  performance  with  decision-theoretic  aids  has  been  a 
subject  of  considerable  controversy  in  recent  years.  The  debate  has  re¬ 
volved  around  the  relevance  or  irrelevance  of  outcome  measures,  and  the 
adequacy  or  inadequacy  of  consistency  measures.  The  position  one  adopts 
with  respect  to  these  issues  serves  to  define  a  set  of  empirical  ques¬ 
tions  which  an  evaluation  can  and/or  should  address. 

Performance  measures  of  decision  making  behavior  may  be  derived  on  the 
basis  of  either  external  or  internal  criteria.  External  criteria  are 
defined  in  terms  of  environmental  events  (outcomes)  which  are  not  under 
the  direct  control  of  the  decision  maker.  These  criteria  are  concerned 
with  the  success  or  failure  of  each  decision  in  terms  of  the  "true" 
state  of  the  world.  For  example,  faced  with  an  uncertain  military  si¬ 
tuation,  a  commander  might  decide  to  prepare  for  an  attack.  If,  in 
fact,  the  enemy  was  planning  to  attack,  he  would  be  credited  with  a  suc¬ 
cessful  decision.  The  sole  use  of  outcome  measures  to  define  decision 
making  quality  has  been  criticized  on  the  ground  that  the  decision  maker 
exercises  no  control  over  the  probablistic  outcome  of  the  decisions. 

Internal  criteria,  on  the  other  hand,  do  not  depend  upon  decision  out¬ 
comes,  but  rather  upon  the  degree  of  consistency  between  expressed 
values  and  subsequent  choice  behavior.  If  the  military  commander  in  the 
previous  example  believed  that  the  likelihood  of  an  attack  was  suffi¬ 
ciently  low  and  that  the  cost  of  preparation  was  sufficiently  high,  he 
would  have  been  wise  to  counsel  against  preparatory  action.  The  use  of 
an  internally  consistent  aid  would  have  recommended  this  course  of  ac- 
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tion  and,  although  the  outcome  may  have  been  disastrous,  the  decision 
maker  would  be  judged  to  have  acted  appropriately  in  terms  of  the  re¬ 
quirements  of  normative  decision  theory.  Environmental  outcomes  are  not 
relevant  to  judging  the  adequacy  of  the  decision  maker's  choice  within 
the  context  of  internal  consistency  criteria. 

Furthermore,  the  decision  to  adopt  a  decision-theoretic  aid  should  be 
made  by  a  prospective  user  who  is  informed  of  the  specific  type  of  aid¬ 
ing  he  will  receive.  If  internal  consistency  is  of  primary  importance 
to  the  user,  it  becomes  a  matter  of  secondary  concern  as  to  whether  con¬ 
sistency  is  correlated  with  external  "success."  A  different  user,  how¬ 
ever,  may  justifiably  adopt  the  position  that  "aiding"  is  only  useful 
when  it  serves  to  enhance  the  probability  of  a  successful  outcome.  De¬ 
cision  making  is,  after  all,  the  attempt  to  deal  successfully  with  an 
uncertain  world.  If  the  aiding  system  does  not  further  the  effort,  it 
may  be  viewed  as  superfluous. 

Although  a  number  of  decision-analytic  aids  have  been  developed  toward 
improving  the  decision  making  performance  of  upper  echelon  military  com¬ 
mand  personnel,  few  controlled  studies  have  been  undertaken  to  demon¬ 
strate  their  actual  effect  on  performance.  Most  of  the  "evaluations" 
that  have  been  conducted  have  focused  on  case  studies  of  user  acceptance 
and  human  factors  issues  (e.g..  Sage  and  White,  1980),  and  the  theoreti¬ 
cal  value  of  decision  analysis  (e.g..  Brown  and  Watson,  1976).  One  sig¬ 
nificant  exception  is  a  controlled  experiment  performed  by  Gettys,  May, 
and  0‘Bar  (1976):  although  well- intended ,  difficulties  in  design  and 
methodology  unfortunately  clouded  the  validity  of  its  results.  In  gen¬ 
eral,  therefore,  a  pressing  need  exists  for  convincing  empirical  evalua¬ 
tions  that  identify  the  areas  in  which  the  decision-theoretical  approach 
to  military  decision  making  is  likely  to  have  the  largest  payoff.  These 
evaluations--given  positive  resul ts--would  of  course  also  serve  to  im- 
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press  potential  users  of  the  practical  value  of  decision  aids;  a  good 
example  of  such  an  evaluation  was  recently  performed  by  Siegel  and  Mad¬ 
den  (1980). 


A  decision-analytic  aiding  system  is,  by  definition,  designed  to  func¬ 
tion  in  a  limited  class  of  decision  making  environments.  Therefore,  an 
empirical  evaluation  of  aiding  effectiveness  must  be  conducted  in  accor¬ 
dance  with  a  relevant  set  of  task  requirements.  For  example,  an  aid 
designed  to  assist  a  decision  maker  in  identifying  a  best  course  of  ac¬ 
tion,  given  a  stationary  military  situation  (e.g.,  enemy  intent  is  per¬ 
ceived  to  remain  constant),  cannot  be  appropriately  evaluated  in  the 
context  of  a  non- stationary  environment.  This  incongruence,  incidental¬ 
ly,  was  one  of  the  methodological  difficulties  encountered  in  the 
evaluation  performed  by  Gettys,  et  al ,  (1976).  An  ideal  scenario  is 
thus  one  that  both  lends  itself  to  the  decision-analytic  framework  and 
yet  maintains  a  high  degree  of  task  realism.  Furthermore,  the  scenario 
should  be  such  that  the  empirical  research  findings  can  be  generalized 
to  other,  similar  environments  (Slovic,  Fischhoff,  and  Lichtenstein, 
1977). 

The  conclusion  to  be  drawn  from  the  analysis  of  issues  above  is  that 
evaluative  studies  of  decision-analytic  aids  should  assess  external  as 
well  as  internal  performance  criteria  in  assessing  performance.  Some  of 
the  principal  experimental  questions  that  should  be  addressed  are:  Does 
decision-analytic  aiding  improve  both  the  internal  consistency  and 
external  validity  of  decision-making  performance?  Does  employment  of  a 
decision  aid  increase  user  confidence  in  decisions?  Can  the  aid  be  gen¬ 
eralized  over  different  decision  situations  (i.e.,  scenarios)?  This 
kind  of  research  strategy  was  adopted  here  for  the  evaluation  of  a  sim¬ 
ple,  interactive  decision-analytic  aid. 
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Of  course,  an  operational  evaluation  of  a  decision-analytic  aid  should 
also  address  the  issue  of  user  acceptance.  However,  since  the  recommen¬ 
dations  issued  by  an  aiding  system  can  be  thought  of  as  a  form  of 
"machine"  behavior,  any  system  has  little  to  offer  a  prospective  user 
unless  its  recommendations  are  superior  to  those  provided  by  unaided  hu¬ 
man  performance.  The  issues  of  aiding  effectiveness  and  user  acceptance 
are,  therefore,  logically  and  empirically  independent.  In  other  words, 
user  acceptance  may  be  considered  a  necessary  but  not  sufficient  reason 
for  implementing  a  decision  aid  into  an  actual  command  and  control  sys¬ 
tem. 

Research  Objectives 

Because  of  the  difficulty  in  defining  an  external  criterion  of  decision 
making  performanced ,  previous  studies  have  typically  opted  to  evaluate 
decision  aiding  in  terms  of  internal  criteria.  In  this  research,  how¬ 
ever,  an  attempt  was  made  at  the  outset  to  define  an  explicit  external 
criterion  of  decision  making  performance.  Thus,  the  overall  objective 
was  to  determine  whether  decision  makers  using  a  particular  decision  aid 
select  "better"  decision  alternatives  than  unaided  analysts  in  situa¬ 
tions  where  decisions  could  be  evaluated  according  to  an  established  1 
outcome  criterion  or  "ground  truth."  The  research  consisted  of  prelim¬ 
inary  studies  and  a  main  experiment.  The  preliminary  studies  were 
designed  to  develop,  refine,  and  validate  a  number  of  scenarios;  these 
efforts  culminated  in  a  scenario  validation  study  designed  to  establish 
a  clearly  specified  resultant  state-of-the-world  for  each  scenario  ver¬ 
sion.  These  ground-truth  bases  were  used  to  evaluate  performance  in  the 
subsequent  main  experiment. 


The  main  experiment,  referred  to  as  the  decision  making  study,  was 
designed  to  compare  the  quality  of  decision  making  performance  under  two 


conditions.  Under  the  first,  or  experimental  condition,  subjects  were 
led  through  a  complete  decision  analysis  using  a  standard,  computer- 
supported  decision  aiding  package;  under  the  second,  or  control,  condi¬ 
tion,  subjects  were  required  to  make  decisions  without  the  benefit  of 
decision  aiding.  The  decision  making  performance  of  both  groups  was 
compared  according  to  the  ground- truth  criterion  established  in  the 
prel iminary  studies. 

Target  of  Evaluation 

Several  general  purpose  decision  analysis  software  packages  have  been 
developed  by  various  organizations.  The  target  of  the  present  evalua¬ 
tion  is  called  Rapid  Screening  of  Decision  Options  (OPINT)  developed  by 
Decisions  and  Designs,  Inc.  (DDI)--see  Selvidge,  1976.  This  system  was 
designed  to  aid  individual  decision  making  and  although  simplified  in 
scope,  it  is  composed  of  procedures  representative  of  those  used  in  oth¬ 
er  decision  analytic  systems.  The  package  contains  models  for  probabil¬ 
ity  influence,  Bayesian  revision,  multi-attribute  utility,  and  subjec¬ 
tive  expected  utility  (SEU)  aggregation  which  combine  to  form  a  logical¬ 
ly  integrated  decision  support  system.  The  package  has  been  implemented 
on  an  IBM-5110  (programmed  in  APL)  and  operates  in  a  standalone,  turn¬ 
key  fashion.  The  components  of  the  system  are  illustrated  in  Table  1. 
Each  component  consists  of  a  program  module  which  serves  a  different 
aiding  function.  The  information  required  of  the  user  (elicited  input) 
and  that  provided  by  the  aid  (output)  are  specified  separately  for  each 
module. 

Probability  Influence  Model.  An  important  stage  of  decision  analysis 
requires  an  individual  to  estimate  the  probabil ities  associated  with 
each  possible  state  of  the  world.  Whenever  these  probabilities  are  con¬ 
ditionally  dependent  (i.e.,  under  the  influence  of  intervening  events), 
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TABLE  1 


OPINT  MODULES 


PROGRAM  MODULE 

ELICITED  INPUT 

OUTPUT 

Probability  Influence 
Model 

Conditional 

Probabilities 

Prior  Probabilities 

For  Outcomes 

Bayesian  Revision 

Model 

Likelihood  Ratios 

Posterior  Probabilities 
For  Outcomes 

(II) 

MAU  Model 

Decomposed  Value 
Judgments 

Utilities  for  Decision 
Alternatives 

(III) 

Sensitivity  Analysis 

Decomposed  Value 
Judgments,  Output 
III 

Change  in  SEU  as  a  Function 
of  Various  Outcome-Proba-  (IV) 
bility  Levels 

SEU  Aggregation 

Model 

Outputs  II  and  III 

SEU  for  Decision  Alter¬ 
natives  with  Recommenda¬ 
tion  for  Best  Choice 

(V) 

the  probability  influence  model  may  be  used  to  appropriately  decompose 
the  probability  elicitation  procedure.  This  program  queries  the  user  as 
to  his  understanding  (i.e.,  his  "model")  of  the  event  chain  in  question. 
Given  a  pattern  of  dependent  relationships,  the  program  then  requires 
conditional  probabilities  which  it  uses  to  compute  joint  probabilities. 
Finally,  these  are  aggregated  to  produce  a  set  of  unconditional  proba¬ 
bilities  associated  with  the  main  uncertain  events. 

Bayesian  Revision  Model.  When  presented  with  new  information,  the  deci¬ 
sion  maker  may  wish  to  modify  his  initial  (prior)  probability  estimates. 
The  Bayesian  revision  model  enables  the  decision  maker  to  update  his 
probabilities  in  accordance  with  the  normative  model.  The  program  eli- 
cites  likelihood  ratios  (P (D  |  H ) )  which  it  uses  to  compute  posterior  pro¬ 
babilities  (P (H  | D ) ) .  These  revised  estimated  form  the  basis  for  modi¬ 
fied  expected  utility  calculations. 

Multi-Attribute  Utility  Model.  The  mul ti-attribute  utility  (MAU)  model 
provides  the  decision  maker  with  an  opportunity  to  systematically  evalu¬ 
ate  relevant  problem  dimensions.  Each  decision  alternative  is  decom¬ 
posed  into  a  number  of  attribute  (value)  dimensions  which  are  assessed 
individually.  Following  this  elicitation  procedure,  the  MAU  program  au¬ 
tomatically  multiplies  each  attribute  level  by  its  assigned  weight  and 
calculates  the  sum  of  these  products.  Ihe  result  is  a  measure  of 
overall  utility  for  each  decision  alternative. 

Sensitivity  Analysis.  The  sensitivity  analysis  module  provides  a  simple 
and  rapid  procedure  for  allowing  the  decision  maker  the  option  to  see 
how  his  utilities  are  calibrated,  so  that  any  necessary  refinements  can 
be  made.  The  program  utilizes  the  parameters  of  the  MAU  model  (decom¬ 
posed  value  judgments,  attribute  weights,  computed  utilities  for  deci¬ 
sion  alternatives)  to  provide  a  matrix  showing  how  the  relative  subjec- 
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tive  expected  utility  (SEU)  of  each  decision  alternative  changes  with 

different  probabilities  (0.00,  .10,  .20 . 90,  1.00)  for  a  given 

outcome  (i.e.,  state-of-the-world) .  By  making  changes  in  his 
attribute-importance  weights  or  any  of  his  single-attribute  utility  ma¬ 
trices,  the  subject  can  observe  the  sensitivity  of  the  profile  of  recom¬ 
mended  decision  alternatives  and  adapt  the  component  parameters  accord¬ 
ingly. 

SEU  Aggregation.  The  final  stage  of  the  decision  making  process  in¬ 
volves  the  integration  of  available  information  and  the  selection  of  a 
single  alternative.  The  programmed  aid  performs  this  function  automati¬ 
cally  and  displays  a  recommended  course  of  action.  A  subjective  expect¬ 
ed  utility  (SEU)  aggregation  model  serves  to  integrate  the  decision 
maker's  subjective  inputs  (probabilities  and  utilities)  with  an  inter¬ 
nally  consistent  theory  of  choice  behavior.  The  aid  then  identifies  the 
alternative  associated  with  the  highest  SEU  and  displays  it  to  the  deci¬ 
sion  maker  in  the  form  of  a  recommendation. 

In  summary,  then,  the  0PINT  decision  aiding  system  performs  two  primary 
functions.  It  serves  to  provide  structure  to  the  decision  making  en¬ 
vironment  by  forcing  the  decision  maker  to  consider  relevant  dimensions 
of  the  problem  (e.g.,  the  probability  influence  and  MAU  models).  In  ad¬ 
dition,  the  aid  aggregates  subjective  input  by  computing  probability 
values  and  offering  decision  recommendations  in  accordance  with  a  norma¬ 
tive  model  (e.g.,  the  Bayesian  revision  and  SEU  aggregation  models).  In 
essence,  task  structure  is  provided  indirectly  by  elicitation  pro¬ 
cedures,  while  the  results  of  aggregation  are  displayed  directly  (cf. 
Brown  and  Watson,  1976);  either  or  both  of  these  directions  may  be 
necessary  to  achieve  successful  decision  aiding. 


DDI  has  demonstrated  the  operational  capability  of  this  aiding  system  to 
assist  in  decision  making  in  a  number  of  military  problems  (e.g.,  Warsaw 
Pact  scenario).  According  to  Selvidge  (1976),  the  reactions  of  users 
have  been  positive  overall;  she  states: 


"The  solutions  to  problems.. .are  seen  as  plausible  by  the 
users  of  the  method  in  light  of  their  explicit  probability 
and  value  assessments.  Furthermore,  the  discussion  of 
these  probabilities  and  values  has  improved  communication 
among  different  parties  to  the  decision.  The  users  are 
also  enthusiastic  about  their  ability  to  modify  by  them¬ 
selves  both  the  structure  of  the  problem  and  its  inputs" 
(p.  64). 


These  remarks  support  the  utility  of  this  aiding  package  in  terms  of 
user  acceptance;  however,  controlled  empirical  evaluations  are  required 
to  assess  the  aid's  functional  capability  to  significantly  improve  deci¬ 
sion  performance.  With  this  purpose  in  mind,  the  present  research  was 
conducted. 
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PRELIMINARY  STUDIES 


Before  undertaking  the  primary  experimental  study  of  decision  aiding,  it 
was  necessary  to  perform  two  preliminary  studies,  one  having  to  do  with 
scenario  construction  and  the  other  with  scenario  validation. 

Scenario  Construction 


Two  scenarios  were  created  to  provide  decision-making  contexts  for  in¬ 
vestigation.  In  order  to  study  the  impact  of  the  decision  maker's 
knowledge  with  respect  to  background  context,  one  pair  of  scenarios  was 
based  on  a  "familiar"  situation  and  another  pair  was  based  on  an  "unfam¬ 
iliar"  situation.  The  familiar  scenario,  referred  to  as  the  Balkan 
scenario,  describes  "real-world"  figures  (e.g.,  Marshall  Tito)  and 
"real-world"  countries  (e.g.,  Yugoslavia,  Bulgaria,  etc.).  The  unfami¬ 
liar  scenarios,  referred  to  as  the  Shamba  scenario,  describes  fictitious 
figures  and  countries  (e.g.,  Shamba).  The  latter  background  was  set  in 
the  framework  of  the  Tactical  Negotiations  Game  (Streufert,  Castore,  and 
Kilger,  1967). 

The  background  information  for  the  Balkan  scenario,  given  in  Appendix  A, 
describes  a  plausible  power  struggle  between  liberal  technocrats  and 
hard-line  militants  in  Yugoslavia  following  the  death  of  President  Tito. 
The  USSR  capitalizes  on  the  situation  to  demand  from  Yugoslavia  the 
right  to  create  a  naval  base  in  Montenego.  Since  the  situation  involves 
Soviet  military  interests  and  because  Soviet  nationals  are  in  some 
danger,  there  is  a  possibility  of  a  Warsaw  Pact  attack  from  Bulgaria 
where  Pact  exercises  are  being  held.  The  intelligence  analyst 
represents  the  U.S.  interests,  and  the  Warsaw  Pact  is  considered  the 
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enemy.  Each  message  used  in  the  scenario  describes  either  a  decrease  or 
increase  in  enemy  political  and  military  belligerence. 

The  background  information  for  the  Shamba  scenario,  given  in  Appendix  B, 
describes  a  small  Asian- like  developing  country,  called  Shamba,  beset  by 
internal  revolution  and  foreign  intervention.  The  intelligence  analyst 
represents  the  Union  of  North  Hemispheric  States  (UNHS),  a  powerful 
country  with  democratic  ideals  committed  to  preserving  freedom  in  Sham¬ 
ba.  The  UNHS  supports  a  military  government  in  Shamba  against  a  rebel 
movement  (i.e.,  the  enemy)  supported  by  a  number  of  foreign  powers.  The 
scenario  requires  the  analyst  to  determine  whether  the  rebels  and  their 
supporters  intend  to  launch  a  major  offensive  in  the  near  future.  As 
with  the  Balkan  scenario,  each  message  reflects  either  an  increase  or 
decrease  in  enemy  belligerence. 

For  both  the  Balkan  and  Shamba  settings,  two  scenario  versions  (i.e.,  a 
pair  of  scenarios)  were  developed  to  reflect  each  of  two  possible  states 
of  the  world,  namely  the  enemy's  intention  to  attack  and  the  enemy's  in¬ 
tention  not  to  attack.  The  attack  (A)  and  no  attack  (NA)  versions  of 
the  Balkan  and  Shamba  scenarios  each  consist  of  a  set  of  five  separate 
intelligence  summaries,  with  five  different  messages  contained  in  each 
summary.  Thus,  a  total  of  four  different  message  scenarios  were 
employed--an  A  and  NA  version  for  the  Balkan  context,  and  an  A  and  NA 
version  for  the  Shamba  context. 

To  allow  for  the  comparability  of  decision  making  performance  between 
familiar  and  unfamiliar  contexts,  the  background  descriptions  for  the 
Balkan  and  Shamba  settings  were  made  as  parallel  as  possible.  Then, 
messages  were  created  in  semantically  related  pairs,  one  for  each 
scenario;  that  is,  an  attempt  was  made  to  construct  the  members  of  each 
message  pair  as  similar  as  possible  in  both  content  and  meaning.  An  ex- 
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ample  of  a  Balkan  message  and  its  Shamba  counterpart  are  as  follows: 

Balkan:  The  Soviets  are  pushing  for  a  summit  meeting  this 
year  to  finalize  the  signing  of  a  SALT  II  pact 
and  to  outline  the  next  steps  in  detente. 

Shamba:  The  Socialist  Alliance  is  pushing  for  a  summit 
with  the  UNHS  this  year  to  sign  a  treaty  which 
will  mutually  limit  the  number  and  types  of 
weapon  systems  sold  to  developing  countries. 

A  pool  of  75  pairs  of  Balkan  (B)  and  Shamba  (S)  messages  were  prepared 
for  initial  evaluation.  The  messages  in  a  given  pair  (i)  can  be  re- 
ferrea  to  as  BM^  and  SM^  ,  respectively. 

The  background  information  and  message  sets  actually  used  in  the  deci¬ 
sion  aiding  study  were  obtained  using  a  three-phase  selection  and  re¬ 
finement  process.  The  first  step  involved  a  study  of  the  diagnostic  im¬ 
pact  of  each  individual  message.  After  being  trained  to  elicit  condi¬ 
tional  probability  assessments,  eight  senior  reserve  naval  intelligence 
officers  were  presented  with  the  various  scenario  situations  (one  at  a 
time)  and  were  asked  to  judge  the  likelihood  of  the  occurrence  of  the 
event(s)  described  in  each  individual  message  given  the  indicated  state 
of  the  world.  First,  they  were  provided  a  written  description  of  the 
Balkan  background  data  and  were  then  told  to  assume  that  the  enemy  in¬ 
tended  to  attack.  Given  this  assumption,  they  were  asked  to  judge  the 
conditional  probability,  P(BM.jA)  for  each  of  75  messages.  Next,  given 
the  same  Balkan  background,  they  were  told  to  assume  that  the  enemy  did 
not  intend  to  attack  and  were  asked  to  judge  P(BM.|nA)  for  each  message. 
Then  the  same  procedure  was  repeated  for  the  Shamba  scenario:  after  be¬ 
ing  presented  with  the  common  background  information,  likelihood  esti- 
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mates  P (SM^  j A)  were  first  obtained  given  an  intended  enemy  attack,  and 
then  the  complementary  estimates,  P(SMi | NA ) ,  were  elicited  given  no  at¬ 
tack;  again,  the  index  i  moved  from  1  to  75  to  cover  all  the  Shamba  mes¬ 
sages. 

The  elicitation  data  were  reduced  across  the  eight  subjects  to  provide  a 
median  conditional  probability  judgment  within  each  situation.  That  is, 
for  every  message  (i),  the  following  measures  were  obtained: 

Pmed(BMilA)>  W^ilM)  or  pmed(SMiiA)>  Pmed(SMi lNA)*  A  likelihood 
ratio  (LR)  was  then  derived  for  each  message  to  provide 


LR(BMi ) 


pmed  (Bp*i  I  A^ 

T^TBMTfNA)  • 


for  the  Balkan  scenario,  and 


LR  (SM 


'>  pmed(SHi| HA)  • 


for  the  Shamba  scenario.  To  enhance  message  similarity  across  the  dif¬ 
ferent  scenario  contexts,  these  likelihood  ratios  were  used  as  a  basis 
for  deleting  all  message  pairs  that  did  not  both  favor  the  same  state  of 
the  world  (either  attack  or  no  attack).  That  is,  only  the  message  (i) 
pairs  where  both  likelihood  ratios  across  the  two  scenarios  were  greater 
than  one  (LR(BM.)  >  l,  and  LR(SM^)  >1  ),  or  less  than  one  (LR(BM1)  <  1, 
and  LR(SM^)  <  l  )  were  retained  in  the  working  message  pool.  Further¬ 
more,  to  exclude  messages  with  extremely  high  diagnosticity ,  only  pairs 
with  1/3  <  LR(BMi)  <  3  and  1/3  <  LR(SM^)  <  3  were  retained  for  further 
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consideration. 


In  the  second  phase  of  scenario  development,  the  message  pairs  (consist¬ 
ing  of  one  Balkan  and  one  Shamba  message  with  likelihood  ratios  LR ( BM^ ) 
and  LR(SM^),  respectively)  were  selectively  sampled  (without  replace¬ 
ment)  to  obtain  specific  sets  of  message  pairs  that  indicated  attack  and 
no-attack  states  of  the  world.  Each  set,  called  an  intelligence  sum¬ 
mary,  was  made  up  of  exactly  five  messages.  A  total  of  16  summaries 
were  constructed,  or  four  for  each  scenario  version;  these  summaries  are 
labeled  in  accordance  with  their  scenario  membership  as  follows:  BAS-- 

J 

-Balkan  attack  summary,  BNASj  — Ba 1 kan  no  attack  summary,  SASj--Shamba 
attack  summary,  and  SNASj--Shamba  no  attack  summary. 

To  obtain  the  attack  versions  of  each  scenario,  successive  summaries 
were  configured  so  that:  (a)  the  number  of  messages  out  of  five  which 
favored  an  "attack"  (i.e.,  LR(BMi)  >  l,  and  LR(SMi)  >  1)  equaled  3,  3, 
4,  and  4  for  summaries  one  through  four,  respectively;  and,  (b)  the  cu¬ 
mulative  product  of  the  five  likelihood  ratios  within  a  summary  fell 
between  1.0  and  1.5,  1.5  and  2.5,  2.5  and  3.5,  and  1.5  and  2.0  for  sum¬ 
maries  one  through  four,  respectively.  This  set  of  constraints  was 
selected  to  systematically  increase  the  running  posterior  likelihood  ..ra¬ 
tio  in  favor  of  an  attack  from  an  initial  value  of  one  to  a  terminal 
value  of  about  ten,  as  a  function  of  the  cumulative  information  con¬ 
tained  in  the  successive  intelligence  summaries. 

The  identical  process  was  employed  to  derive  the  message  sets  for  the  no 
attack  versions  of  the  scenarios.  First,  however,  for  the  message  pairs 
remaining  in  the  pool  after  the  attack-scenario  messages  were  removed, 
the  likelihood  ratios  were  inverted  so  that  a  ratio  greater  than  one  im¬ 
plies  that  the  no  attack  state  of  the  world  is  favored  (i.e.,  is  more 
probable).  Then  the  same  selection  constraints  applied  to  the  likeli- 
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hood  ratios  with  A  in  the  numerator  and  NA  in  the  denominator  were  now 
applied  to  the  likelihood  ratios  with  NA  in  the  numerator  and  A  in  the 
denominator.  That  is,  the  derivation  and  statistical  description  of 
messages  for  the  no  attack  scenario  versions  are  exactly  as  in  the  pre¬ 
vious  paragraph  except  that  each  occurrence  of  the  term  "attack"  would 
be  replaced  with  "no  attack."  For  example,  if  3  messages  out  of  5 
favored  an  attack  in  summaries  BAS^  and  SAS-,  then  3  out  of  5  favored  no 
attack  in  summaries  BNASj  and  SNASj .  Hence,  as  a  result  of  the  message 
selection  process,  a  total  of  four  preliminary  message  scenarios  (each 
consisting  of  20  different  messages  or  four  sets  of  five  messages  each) 
were  assembled  to  represent  an  A  and  NA  version  for  the  Balkan  context 
and  a  complementary  A  and  NA  version  for  the  Shamba  context. 

In  the  third  phase  of  development,  the  background  information  and  mes¬ 
sages  for  the  Balkan  scenario  were  reviewed  and  analyzed  for  real-world 
fidelity  by  two  experienced  intelligence  analysts  at  the  CIA,  one  of 
whom  was  a  subject  matter  expert  on  Balkan  affairs.  On  the  basis  of 
their  input,  the  semantic  content  of  the  background  data  and  the  mes¬ 
sages  were  modified  and  improved  to  more  accurately  reflect  the  actual 
Balkan  situation.  Moreover,  a  fifth  set  of  five  new  messages  was  creat¬ 
ed  for  both  the  A  and  NA  scenarios  to  reflect  as  unambiguously  as  possi¬ 
ble  an  attack  or  a  no-attack  state  of  the  world.  The  Shamba  intelli¬ 
gence  summaries  were  then  modified  to  mirror  the  changes  effected  in  the 
Balkan  scenario  and  a  fifth  intelligence  summary,  designed  to  unambigu¬ 
ously  reflect  the  given  state  of  the  world,  was  also  created  for  this 
scenario. 

Scenario  Val idation 


The  primary  objective  of  the  scenario  validation  study  was  to  determine 
whether  the  two  versions  (A  and  NA)  of  each  scenario  accurately  reflect- 


ed  the  states-of-the-world  they  were  intended  to  convey.  The  construc¬ 
tion  of  the  message  scenarios  was  based  on  mathematical  combinations  of 
individual  likelihood  ratios;  that  is,  the  intelligence  summaries  were 
configured  without  taking  into  account  the  possible  conditional  nonin¬ 
dependence  of  separate  messages  (i.e.,  that  the  joint  diagnostic  impact 
of  two  or  more  messages  taken  together  may  be  very  different  than  the 
result  of  the  numerical  aggregation  of  their  independent  impacts). 

Hence  the  validation  study  was  designed  to  collect  likelihood  ratio  data 
for  each  intelligence  summary  (collection  of  five  messages)  rather  than 
for  each  individual  message. 

Method 

Subjects.  Two  groups  of  reserve  naval  intelligence  officers  consisting 
of  eight  and  nine  officers,  respectively,  participated  in  the  validation 
study.  The  officers'  military  rank  ranged  from  Lieutenant  Commander  to 
Captain,  and  they  had  an  average  of  13  years  career  experience  in  intel¬ 
ligence  analysis.  " 

Design  and  Procedure.  One  group  of  analysts  evaluated  the  A  and  NA  ver¬ 
sions  of  the  Balkan  scenario,  and  the  other  group  evaluated  the  A  and  NA 
versions  of  the  Shamba  scenario.  Analysts  in  both  groups  were  first 
trained  in  understanding  and  making  conditional  probability  assessments 
and  then  were  given  some  practice  trials  with  respect  to  some  real-world 
events.  The  analysts'  practice  conditional  probability  estimates  and 
the  reasons  the  analysts  gave  in  support  of  their  estimates  were  dis¬ 
cussed  by  the  group. 

Once  the  training  procedures  were  completed,  analysts  read  the  back¬ 
ground  information  of  the  appropriate  scenario.  On  the  basis  of  this 
background  information  alone,  they  were  asked  to  estimate  the  a  priori 


probabilities  for  an  enemy  attack,  P ( A) ,  and  no  attack,  P(NA).  Next, 
analysts  were  asked  to  read  each  A  intelligence  summary  (j)  and  to  esti¬ 
mate,  after  each  one,  the  conditional  probability  that  the  set  of  events 
described  by  the  summary  would  occur  given  that  the  enemy  intended  to 
attack,  P(BASj|a)  or  P(SASj ! A) .  Analysts  were  later  asked  to  re-read 
each  A  summary  and,  after  each,  to  estimate  the  conditional  probability 
that  the  events  described  would  occur  given  that  the  enemy  did  not  in¬ 
tend  to  attack,  P ( BAS ^ | na )  or  P(SASj|NA).  The  complementary  procedure 
was  then  used  to  obtain  conditional  probability  estimates  for  the  NA  in¬ 
telligence  summaries.  Probability  estimates  were  recorded  by  the 
analysts  on  specially  prepared  answer  sheets. 

Results  and  Discussion 


The  intelligence  analysts'  mean  probability  estimate  of  an  enemy  attack, 
on  the  basis  of  the  background  information  only,  was  .64  (SD  =  .21)  for 
the  Balkan  scenario  and  .56  (SD  =  .29)  for  the  Shamba  scenario;  and  the 
difference  between  the  two  means,  evaluated  by  a  "t"  test  is  not  sta¬ 
tistically  reliable  (  £  >  .05).  These  data  support  the  assumption  that 
the  scenario  backgrounds  are  relatively  comparable  in  leading  to  a  gen¬ 
erally  uncertain  inference  about  enemy  intentions.  In  other  words,  in 
the  case  of  each  scenario,  the  prior  probabil ities  for  attack  and  no¬ 
attack  are  not  very  far  away  from  .50. 

With  regard  to  the  processing  of  the  intelligence  summaries,  each 
analyst  generated  20  conditional  probability  estimates  as  follows:  Bal¬ 
kan  scenario--  P ( BAS •  j  A )  and  P ( BAS  •  j  NA ) ,  plus  P (BNASi  |  A)  and  P ( BNAS i | NA ) 
for  summaries  (j)  1  through  5;  or,  P(SASj  |a)  and  P (SASj  |NA) ,  plus 
P (SNASj  Ja)  and  P ( SNASj  | NA )  for  summaries  (j)  1  through  5.  These  data 
were  converted  into  group-based  likelihood  ratios  from  median  estimates 
by  the  same  method  as  described  above  for  individual  message  likelihood 
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ratios.  For  example,  for  summary  j  in  the  Balkan  attack-scenario, 

Pmed(  BABj  |  A)  and  Pme<j  ( BAS  j  |  NA )  were  first  obtained  across  the  group  of 
subjects;  then,  a  likelihood  ratio  in  favor  of  an  attack  was  computed  by 
dividing  the  two  median  conditional  probabilities: 


i DfRic  \  _  Pmed(BASjl A) 

LR(6ASJ>  -  • 


For  the  Balkan  no-attack  scenario,  different  intelligence  summaries  are 
involved  and  the  likelihood  ratio  is  inverted,  hence: 


IPfm..  ,  Pmed!BNASi|NA) 
LR(EKASj>-  '^(BNASjiA)  • 


Likewise,  the  same  mathematics  were  applied  to  obtain  comparable  likeli¬ 
hood  ratio  estimates  for  the  Shamba  scenario. 

The  aggregate  results  of  these  analyses  are  displayed  in  Table  2  accord¬ 
ing  to  scenario  version  and  intelligence  summary.  Except  for  the  first 
summary  in  the  Balkan  NA  scenario,  all  likelihood  ratios  are  greater 
than  one,  indicating  that  the  intelligence  summaries  are  perceived  to 
reflect  the  intended  states-of-the-world.  Furthermore,  the  likelihood 
ratios  generally  increase  monotonical ly  across  the  summaries  suggesting 
that,  as  additional  diagnostic  information  is  received,  the  perceived 
odds  in  favor  of  the  "correct"  state-of-the-world  show  a  corresponding 
increase.  The  odds  reach  a  final  level  as  high  as  19:1  (Shamba  A 


TABLE  2 

GROUP-BASED  LIKELIHOOD  RATIOS 


CrCWADTft 

INTELLIGENCE  SUMMARIES 

j  wLIinK  1  u 

1 

2 

3 

4 

5 

Balkan 

Attack  (A) 

1.11 

1.63 

2.64 

3.22 

6.20 

No- Attack  (NA) 

.85 

1.00 

1.39 

2.08 

2.33 

Shamba 

Attack  (A) 

4.00 

3.75 

5.67 

5.67 

19.00 

No-Attack  (NA) 

1.17 

2.33 

2.50 

4.00 

9.00 
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scenario);  but  even  in  the  situation  that  appears  to  be  least  diagnostic 
(Balkan  NA  scenario),  the  terminal  odds  in  support  of  the  intended 
ground  truth  are  greater  than  2:1. 

From  an  examination  of  the  numbers  in  Table  2,  it  is  evident  that  the 
intelligence  summaries  in  the  A  versions  of  the  scenarios  are  perceived 
as  much  more  diagnostic  than  those  for  the  NA  versions.  Perhaps,  this 
finding  results  from  the  difference  in  message  content,  with  the  A 
scenario  being  composed  of  more  eventful  salient,  situation-change- 
oriented  messages  than  the  NA  scenario  which  more  or  less  reflects  the 
maintenance  of  the  status-quo.  Furthermore,  the  level  of  intelligence 
summary  diagnosticity  is  considerably  higher  for  the  Shamba  scenarios 
than  for  the  Balkan  scenario.  This  latter  difference  in  inference  judg¬ 
ments  for  the  unfamiliar  and  familiar  scenarios  may  have  been  influenced 
by  external  knowledge  (outside  the  scope  of  the  background  and  messages) 
which  subjects  brought  to  bear  on  the  problem;  that  is,  such  knowledge 
might  have  led  them  to  be  more  conservative  in  assessing  the  impact  of 
summaries  in  the  familiar,  real-world  situation. 

Overall,  the  pattern  of  results  support  the  goals  of  the  validation 
study.  The  background  information  for  both  the  Balkan  and  Shamba 
scenarios  provided  an  uncertain  decision  making  context,  which  is  pro¬ 
gressively  clarified  as  a  function  of  the  information  subsequently 
presented  in  the  intelligence  summaries.  Moreover,  the  data  demonstrate 
that  all  constructed  versions  of  the  scenarios  do  in-fact,  lead  to  diag¬ 
nostic  inferences  about  the  state-of-the-world  which  correspond  one-to- 
one  with  the  predetermined  intentions  of  the  experimenters.  The  trend 
of  these  results  was  seen  to  be  pretty  much  equivalent  across  the  vari¬ 
ous  scenario  versions.  Although  some  systematic  differences  in  the  re¬ 
lative  magnitude  of  the  intelligence-summary  likelihood  ratios  were  ob¬ 
served  among  scenario  versions  (e.g.,  the  ratios  are  considerably  higher 
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in  the  A  than  in  the  NA  scenarios),  these  differences  can  be  interpreted 
to  reflect  different  degrees  of  ground  truth  validity.  That  is,  for  any 
given  intelligence  summary  (1  through  5),  the  higher  the  likelihood  ra¬ 
tios  in  favor  of  the  intended  state-of-the-world,  the  greater  the  esti¬ 
mates  of  validity  of  the  ground-truth  base.  However,  for  the  purpose  of 
establishing  adequate  performance  criteria  for  the  subsequent  experimen¬ 
tal  evaluation,  the  ground- truth  bases  for  all  scenarios  employed  are 
assumed  to  be  sufficiently  valid. 
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DECISION  AIDING  STUDY 


The  issue  in  this  experiment  is  whether  the  use  of  a  decision  aid  by  in¬ 
telligence  analysts  leads  to  a  significantly  greater  number  of  correct 
decisions  than  are  attained  by  unaided  analysts.  The  analysts'  task  was 
to  diagnose  the  situation  and  recommend  one  of  four  levels  of  alert, 
ranging  from  "no  alert"  to  a  "reinforced  alert,"  on  the  basis  of  the 
scenario  background  and  intelligence  summary  information  presented. 

Because  the  intelligence  summaries  were  constructed  to  reflect  a  pre- 
established  state-of-the-world ,  it  is  assumed  that  the  experimenter 
plays  the  role  of  a  prescient  observer  who  knows  the  "correct"  state- 
of-the-world.  Decisions  that  are  congruent  with  the  designated  state- 
of-the-world  are  considered  correct,  and  those  that  are  not  congruent 
are  considered  incorrect.  In  the  current  scenarios,  if  an  analyst  knows 
for  certain  that  the  enemy  is  going  to  attack  (i.e.,  P(A)  =1),  the 
correct  decision  alternative  is  to  place  troops  on  the  highest  level  of 
alert.  Similarly,  if  an  analyst  knows  for  certain  that  the  enemy  is  not 
going  to  attack  (i.e.,  P(NA)  =  1)  the  correct  decision  alternative  is  to 
place  troops  on  the  lowest  level  of  alert.  These  normative  responses, 
of  course,  assume  that  under  an  A  state-of-the-world,  an  analyst  would 
most  prefer  the  highest  level  of  alert--i.e.,  reinforced  alert--and 
under  a  NA  state-of-the-world  an  analyst  would  most  prefer  the  lowest 
level  of  alert--i.e.,  no  alert.  Since  the  attainment  of  correct  deci¬ 
sions  is  a  probabilistic  process,  one  cannot  expect  aided  subjects  to  be 
correct  for  each  and  every  decision.  However,  given  the  objectives  of 
decision  aiding,  it  is  not  unreasonable  to  hypothesize  that  over  the 
long  run,  the  relative  frequency  of  correct  decisions  attained  by  aided 
analysts  should  be  significantly  greater  than  for  unaided  analysts. 


Method 


Subjects.  The  subjects  were  24  reserve  naval  intelligence  officers 
ranging  in  rank  from  Lieutenant  to  Commander,  none  of  whom  participated 
in  the  preliminary  studies.  The  officers  were  attached  to  units  dril¬ 
ling  in  Alameda,  Los  Alamitos,  and  San  Diego,  California.  They  had  an 
average  of  9.8  years  career  experience  in  intelligence  analysis. 

Design.  Subjects  were  randomly  assigned  to  two  groups:  a  fully  aided 
experimental  group  and  an  unaided  control  group.  Subjects  in  both  con¬ 
ditions  were  given,  in  sequence,  one  version  (A  or  NA)  of  the  Balkan 
scenario  and  the  opposite  version  of  the  Shamba  scenario.  The  presenta¬ 
tion  order  of  the  scenarios  was  completely  counterbalanced  and  the  vari¬ 
ous  scenario  combinations  were  used  with  equal  frequency.  Subjects  in 
both  conditions,  aided  and  unaided,  were  given  the  versions  of  the  Bal¬ 
kan  and  Shamba  background  data  and  message  summaries  shown  in  Appendices 
A  and  B  respectively.  The  decision  aid  used  in  the  study  was  Decision 
and  Design ' s  OPINT  decision  aiding  package  (Selvidge,  1976)  programmed 
on  an  IBM  5110  microcomputer,  as  described  above. 

Procedure.  The  12  subjects  in  each  of  the  two  treatment  conditions  were 
informed  that  they  were  participating  in  a  study  of  complex  human  deci¬ 
sion  making,  and  that  their  responses  would  remain  conf identi al --i .e. , 
no  raw  data  would  be  reported.  All  subjects  were  instructed  to  assume 
the  role  of  intelligence  analysts.  Their  purported  mission  was  to 
recommend,  at  the  request  of  their  commanding  officer,  one  of  four  lev¬ 
els  of  alert  on  the  basis  of  available  intelligence  data;  these  were  to 
be  recommendations  made  repeatedly  after  reading  each  of  five  successive 
intelligence  summaries.  The  four  levels  of  alert,  patterned  after  the 
scenario  described  in  Selvidge  (1976),  were  to  place  troops  on:  (a)  no 
alert,  (b)  military  vigilance,  (c)  simple  alert,  and  (d)  reinforced 
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alert.  The  experimenter  explained  that  the  levels  of  alert,  i.e.,  deci¬ 
sion  alternatives,  were  to  be  considered  as  an  equal  interval  scale  of 
increasing  military  preparedness. 

After  indicating  a  preferred  decision  alternative,  the  subject  was  asked 
to  indicate  the  degree  of  confidence  he  had  in  his  selection.  This  was 
done  by  the  assignment  of  100  points  across  the  four  possible  decision 
options,  with  the  chosen  option  of  course  receiving  the  highest  number 
of  points.  So,  for  example,  the  analyst  might  assign  confidence  points 
of  85,  15,  0  and  0  to  decision  alternatives  1  through  4,  respectively. 
Analysts  were  instructed  to  state  their  confidence  vector  as  accurately 
and  honestly  as  possible  since  their  implicit  score  for  each  decision 
would  be  determined  by  not  only  whether  it  was  correct  or  not,  but  also 
how  prudently  they  had  distributed  confidence  among  the  alternatives. 

Table  3  outlines  the  specific  tasks  that  both  aided  and  unaided  subjects 
performed  during  the  course  of  the  experiment.  In  the  aided  condition, 
the  experimenter  played  the  role  of  a  decision  analyst  and  guided  the 
subject  through  the  aiding  procedure.  The  procedure  was  based  on  the 
guidelines  given  by  Selvidge  (1976)  for  using  DDI's  OP  I  NT  model  and  spe¬ 
cial  care  was  taken  not  to  bias  the  subject  toward  the  correct  response. 
The  analysts  were  first  given  the  background  information  for  the  first 
scenario  to  read  and  were  then  instructed  in  the  concept  of  SEU.  Fol¬ 
lowing  this,  they  were  asked  to  estimate  the  unconditional  (i.e.,  "unin¬ 
fluenced")  probabilities  of  an  enemy  A  and  NA  state-of-the-world.  Sub¬ 
jects  were  then  led  through  the  probability  influence  model  of  OPINT, 
which  asked  them  to  consider  antecedent  events  leading  to  an  enemy  A  and 
to  NA  state-of-the-world.  The  model  requires  conditional  subjective 
probability  estimates  for  the  antecedent  events  and  ultimately  calcu¬ 
lates  the  joint  probabil ities  of  A  and  NA.  The  computed  conditional  (or 
"influenced")  probabilities  of  A  and  NA  were  compared  to  the  initial 
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TABLE  3 


TASK 

PROCEDURE 

AIDED  GROUP 

UNAIDED  GROUP 

Step  1: 

Present  scenario  back¬ 

Step  1: 

Present  scenario  back¬ 

ground 

ground 

Step  2: 

Elicit  unconditional 

Step  2: 

Present  first  intelligence 

probability  estimates 

summary 

Step  3: 

Develop  probability 

Step  3: 

Elicit  decision  choice 

Step  4: 

influence  model 

Step  4: 

Elicit  confidence  vector 

Elicit  multi-attribute 
utilities 

Step  5: 

Repeat  steps  2  through  4 
with  intelligence  summaries 

Step  5: 

Perform  sensitivity 
analysis 

two  through  five 

Steo  6: 

Present  first  intelli¬ 

Step  6: 

Repeat  steps  1  through  5 
with  second  scenario 

Step  7: 

gence  surranary 

Step  7: 

Elicit  multi-attribute 

Elicit  conditional 

utilities  and  dimension 

probability  estimates 

weights 

Steo  8: 

Update  state-of-the- 
world  probabilities 
via  Bayesian  revision 
model 

v 

Step  9: 

Show  SEU  values  for 
decision  options 

Step  10: 

Elicit  decision  choice 

Step  11: 

Elicit  confidence  vector 

Step  12: 

Repeat  steps  6  through 

11  with  intelligence 
summaries  two  through 
five 

Step  13: 

Repeat  steps  1  through  12 
with  second  scenario 
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non-conditional  estimates  and  the  subject  adopted  those  probabilities  of 
N  and  NA  judged  most  appropriate. 


The  next  step  in  the  aiding  process  involved  the  use  of  the  multi¬ 
attribute  utility  model  to  elicit  the  subject's  utilities  across  a  ma¬ 
trix  specifying  each  decision-option  by  state-of-the-world  combination 
(one  cell  of  the  matrix  defines,  for  example,  the  combination  of  "rein¬ 
forced  alert"  and  "no  attack").  In  the  OPINT  model,  utilities  are  as¬ 
sessed  in  terms  of  relative  regret.  That  is,  the  subject  is  asked  to 
make  judgments  concerning  how  much  more  regretful  he  would  feel  if  one 
combination  occurred  as  opposed  to  another;  for  example,  given  a  resul¬ 
tant  attack,  if  he  had  a  regret  value  of  -100  for  having  decided  on  “no 
alert,"  how  much  less  would  he  regret  having  decided  on  "military  vigi¬ 
lance."  The  regret  elicitation  process,  however,  was  first  decomposed 
by  having  subjects  enumerate  a  few  dimensions  or  attributes  which  would 
impact  on  the  value  of  each  outcome  combination  (e.g.,,  military  risk, 
alert  preparations,  political  consequences).  Then  a  separate  matrix  was 
elicited  which  contained  regret  values  for  each  outcome  combination  with 
respect  to  each  value  dimension.  Finally,  the  subject  was  asked  to  pro¬ 
vide  weights  as  to  the  relative  importance  of  each  value  dimension. 

These  elicitation  data  were  then  entered  into  the  computer  which  em¬ 
ployed  the  multi-attribute  utility  model  to  generate  a  single  matrix 
containing  the  aggregate  regret  values  for  the  eight  outcome  cells. 

Once  the  regret  judgments,  i.e.,  utilities,  were  obtained,  the  aided  de¬ 
cision  makers  were  shown  the  results  of  a  sensitivity  analysis,  provided 
as  an  option  of  the  OPINT  package.  In  the  sensitivity  analysis,  the 
computer  algorithm  systematically  varies  the  subjective  probability  for 
a  given  state-of-the-world  from  0  to  100  in  increments  of  10  and  calcu¬ 
lates  the  respective  SEU  for  each  decision  alternative.  This  informa¬ 
tion  is  then  displayed  and  explained  by  the  experimenter  so  that  the 
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subject  can  see  how  the  decision  alternatives  compare  to  each  other  in 
attractiveness  (i.e.,  SEU)  for  the  different  probability  values.  Most 
analysts  agreed  that  the  recommended  decision  alternative  as  a  function 
of  the  state-of-the-world  probabilities  reflected  their  actual  decision 
preference  profile.  For  the  few  analysts  who  felt  uncomfortable  with 
the  displayed  preference  profile,  changes  were  made  either  in  the 
analysts'  single-dimensional  regrets  or  the  dimension  weights  to  produce 
a  preference  profile  more  in  line  with  the  analysts'  intuitions. 

After  the  sensitivity  analysis  was  completed,  the  subject  was  given  the 
first  intelligence  summary  to  read  and  was  asked  to  give  conditional 
probabilities  that  the  events  described  would  occur  given  that  the  enemy 
intended  to  attack  (A)  and  given  that  the  enemy  did  not  intend  to  attack 
(NA).  The  data  were  entered  into  the  Bayesian  revision  model  of  the  de¬ 
cision  aid  and  the  subject  was  shown  the  revised  computed  probabilities 
of  A  and  NA.  Next,  the  subject  was  shown  the  SEU  for  each  decision  al¬ 
ternative  and  the  experimenter  explained  which  option  had  the  highest 
SEU.  The  subject  was  then  asked  if  he  accepted  the  recommended  alterna¬ 
tive.  If  not,  the  experimenter  asked  which  other  decision  alternative 
the  analyst  preferred.  Once  the  preferred  alternative  was  chosen,  the 
experimenter  asked  the  subject  to  indicate  his  confidence  vector  across 
all  four  decision  al ternati ves.  The  task  continued  in  that  the  subject 
was  given  the  second  intelligence  summary,  conditional  probabilities 
were  elicited  (Step  7,  Table  3),  etc.  until  all  five  intelligence  sum¬ 
maries  were  analyzed.  Later,  the  entire  procedure  was  repeated  with  the 
second  scenario.  Overall,  about  four  hours  were  required  to  run  an  aid¬ 
ed  subject. 

In  the  unaided  condition,  subjects  were  first  given  the  background  in¬ 
formation  of  the  first  scenario.  This  was  followed  by  the  five  intelli¬ 
gence  summaries.  After  each  one,  the  subject  was  asked  to  recommend  one 
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of  the  four  decision  alternatives  as  well  as  to  indicate  his  confidence 
vector  across  the  alternatives.  Once  the  subject  had  completed  both 
scenarios,  the  experimenter  elicited  utilities  and  dimension  weights  us¬ 
ing  the  same  procedure  employed  with  the  aided  subjects.  The  entire 
procedure  lasted  about  1  1/2  hours  for  each  unaided  subject.  Although 
aided  subjects  performed  for  a  much  longer  period  than  unaided  subjects, 
most  of  the  added  time  was  absorbed  in  the  mechanics  of  the  aiding  pro¬ 
cess;  thus,  both  groups  devoted  roughly  equivalent  time  to  the  reading 
of  intelligence  summaries. 

Results 


Five  different  analyses  are  reported  below.  The  first  three  analyses 
compare  aided  versus  unaided  analysts  in  terms  of:  percent  correct  de¬ 
cisions,  confidence  validity,  and  utility  (regret)  judgements.  The 
fourth  analysis  investigates  the  a  priori  probability  and  likelihood  ra¬ 
tio  estimates  elicited  from  the  aided  subjects.  Finally,  the  fifth 
analysis  compares  the  performance  of  the  aided  analysts  to  the  perfor¬ 
mance  of  the  decision  aid.  This  last  comparison  is  possible  because  the 
aided  analysts  were  not  constrained  to  accept  the  recommendations  of  the 
decision  aid. 

Percent  correct  decisions.  In  the  A  scenarios,  decisions  to  place 
troops  on  a  "reinforced  alert"  are  counted  as  correct;  in  the  NA 
scenarios,  decisions  to  place  troops  on  "no  alert"  are  counted  as 
correct.  The  results  are  reported  in  terms  of  percent  correct  out  of 
five  decisions  per  scenario.  The  main  results,  displayed  in  Figure  1, 
reveal  that  aided  analysts  achieve  substantially  more  correct  decisions 
than  unaided  analysts  under  the  A  scenarios  (67%  versus  20%),  and  that 
unaided  analysts  attain  somewhat  more  correct  decisions  than  aided 
analysts  in  the  NA  scenarios  (40%  versus  10%).  A  four-way  analysis  of 


35 


r 


variance  (ANOVA)  comparing  aiding  to  no  aiding,  A  versus  NA  scenario, 
scenario  combinations,  and  presentation  orders,  indicates  that  the  in¬ 
teraction  shown  in  the  figure  is  statistically  reliable,  £  (1,  16)  = 
17.70,  2  <  .01.  Moreover,  the  main  effect  for  the  A  versus  NA  factor  is 
also  significant,  £  (1,  16)  =  7.67,  2  <  .05.  All  other  comparisons  are 
not  statistically  reliable. 

Analysis  of  the  simple  main  effect  of  the  interaction  shown  in  Figure  1 
indicates  that  the  aided  analysts  attain  significantly  more  correct  de¬ 
cisions  than  the  unaided  analysts  in  the  A  scenarios,  £  (1,  16)  =  31.36, 
2  <  .001.  However,  the  difference  between  aided  and  unaided  analysts  in 
the  NA  scenarios  fails  to  reach  statistical  reliability,  £  (1,  16)  = 
3.31,  .05  <  2  <  -10- 

Table  4  shows  the  manner  in  which  aided  and  unaided  analysts  distributed 
their  decisions  across  the  four  alternatives  under  the  A  and  NA 
scenarios.  On  the  one  hand,  in  the  A  scenarios  the  data  show  that  aided 
analysts  were  far  more  willing  to  adopt  the  highest  level  of  alert-- 
i.e.,  "reinforced  alert"--than  were  the  unaided  analysts.  In  contrast, 
the  unaided  analysts  mostly  preferred  to  recommend  "simple  alert"  and 
"military  vigilance"  in  these  scenarios.  On  the  other  hand,  in  the  NA 
scenarios  the  aided  analysts  distributed  their  decisions  more  evenly 
across  the  alternatives  and  recommended  the  higher  levels  of  alert  in  a 
substantial  number  of  cases.  In  contrast,  the  unaided  analysts  were 
willing  to  adopt  the  two  lowest  levels  of  alert,  viz.,  "no  alert"  and 
"military  vigilance,"  in  a  large  percentage  of  cases. 

If  the  chance  performance  level  for  percent  correct  decisions  is  con¬ 
sidered  to  be  25%  (since  there  were  four  possible  decision  alterna¬ 
tives),  then  actual  performance  by  the  aided  group  in  the  NA  scenarios 
and  by  the  unaided  group  in  the  A  scenarios  appears  to  be  very  poor. 
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TABLE  4 

DISTRIBUTION  OF  DECISION  ALTERNATIVE  SELECTION 


Decision 

Option 

Attack  Scenarios 

No  Attack  Scenarios 

Aided 

Group 

Unaided 

Group 

Aided 

Group 

Unaided 

Group 

No  Alert 

7% 

8% 

10% 

|  40% 

Military  Vigilance 

8% 

25% 

37% 

52% 

Simple  Alert 

18% 

47% 

18% 

8% 

Reinforced  Alert 

[ITT 

20%  ! 

35% 

0% 

Note.  Boxed  values  provide  percent  correct  decisions  for  the  given 
scenario. 
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However,  it  must  be  remembered  that  only  decisions  of  "no  alert"  and 
"reinforced  alert"  were  scored  correct  in  the  NA  and  A  scenarios, 
respectively.  Had  more  relaxed  criteria  been  applibd  such  as  scoring 
either  "no  alert"  or  "military  vigilance"  as  correct  in  the  A  scenario 
and  either  "simple  alert"  or  "reinforced  alert"  as  correct  in  the  NA 
scenario,  then  performance  in  the  two  worst  cases  would  have  been  con¬ 
siderably  higher,  namely,  47%  correct  for  the  aided  group  in  the  NA 
scenarios  and  67%  correct  for  the  unaided  group  in  the  A  scenarios. 
Hence,  part  of  the  poor  performance  showing  for  percent  correct  deci¬ 
sions  in  certain  instances  is  due  to  the  stringent  scoring  criteria  for 
correct  decisions.  Furthermore,  as  the  analysts  obtained  more  informa¬ 
tion,  (i.e.,  as  the  decision  point  was  incremented),  the  level  of  confi 
dence  validity  generally  improved,  and  the  difference  between  aided  and 
unaided  groups  decreased  sharply;  in  fact,  convergence  was  attained  at 
the  final  decision  point. 


Confidence  Val idity.  The  rank  probability  score  (RPS)  (Epstein,  1969), 
a  measure  of  confidence  validity,  was  used  as  an  additional  dependent 
variable  because  it  provides  a  more  sensitive  index  of  the  quality  of 
decision  making  performance  than  the  number  of  correct  decisions.  The 
RPS  is  a  proper  rule  for  scoring  subjective  probabilities  (or  confi¬ 
dences)  that  takes  into  account  the  distribution  of  the  subject's  confi 
dence  across  the  alternatives  as  well  as  the  rank  ordering  of  the  alter 
natives  in  terms  of  correctness;  thus,  the  RPS  is  sensitive  to  the  dis¬ 
tance  of  the  decision  alternative  that  the  analyst  thought  was  correct 
(i.e.,  the  one  on  which  he  placed  his  highest  confidence)  to  the  deci¬ 
sion  altrenative  that  is  actually  scored  as  correct--the  smaller  the 
distance,  the  higher  the  RPS.  Mathematically ,  the  RPS  ranges  from  0  to 
1  with  higher  values  indicating  more  valid  confidence  assessments  in 
light  of  which  alternative  turns  out  to  be  correct;  the  computational 
formula  for  the  RPS  is  somewhat  cumbersome  and  appears  in  Appendix  C. 
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For  example,  within  the  context  of  this  study,  when  the  correct  decision 
alternative  is  "no  alert"  (rank  1),  the  ranking  of  the  other  possible 
choices  in  decreasing  order  of  appropriateness  is  as  follows:  "military 
vigilance"  (2),  "simple  alert"  (3),  and  "reinforced  alert"  (4).  Con¬ 
versely,  when  the  correct  decision  alternative  is  "reinforced  alert" 
(rank  1),  the  ranking  of  the  other  choices,  in  decreasing  order  of  ap¬ 
propriateness,  is:  "simple  alert"  (2),  "military  vigilance"  (3),  and 
"no  alert"  (4).  For  the  confidence  vector  (85,  15,  0,  0),  the  value  of 
the  RPS  is  .99  if  the  first  alternative  is  correct  and  .09  if  the  fourth 
alternative  is  correct. 

The  main  results  of  the  RPS  analysis  are  displayed  in  Figure  2.  In  the 
A  scenarios  (left  panel),  aided  subjects  show  higher  confidence  validity 
than  unaided  subjects;  that  is,  their  confidence  judgments  are  better 
justified  according  to  the  correctness  of  their  decisions.  However,  in 
the  NA  scenarios  (right  panel),  unaided  subjects  provide  more  valid  con¬ 
fidences  than  aided  subjects.  These  results  for  RPS  are  similar  to 
those  obtained  for  the  percent-correct  variable.  The  high  scores  for 
aided  analysts  in  the  A  scenario  (mean  RPS  =  .93)  and  unaided  analysts 
in  the  NA  scenario  (mean  RPS  =  .86)  show  that  when  decision  accuracy  is 
graded  on  a  continuous  scale  by  the  RPS  (rather  than  by  the  all-or-none 
criterion  of  percent  correct  decisions)  analyst  performance  is  reason¬ 
ably  good  relative  to  the  maximum  RPS  score  of  1.0;  as  a  matter  of  fact, 
for  the  remaining  conditions  involving  aided  analysts  in  the  NA 
scenarios  and  unaided  analysts  in  the  A  scenarios,  the  average  RPS  score 
is  .80  which  exceeds  the  chance  RPS  score  of  .71  computed  for  a  confi¬ 
dence  vector  of  (.25,  .25,  25,  25). 

A  preliminary  analysis  of  the  RPS  data  indicated  significant  hetero¬ 
geneity  of  variance  among  experimental  conditions.  Thus,  the  raw  scores 
were  subjected  to  the  following  variance-stabilizing  transformation  pri- 
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or  to  the  performance  of  statistical  analyses: 


Y  = 


where  X  is  the  raw  RPS  and  Y  is  the  transformed  score.  The  data  were 
analyzed  using  a  five-way  hierarchical  ANOVA.  The  five  factors  were: 
aiding  versus  no  aiding,  A  versus  NA  scenarios,  decisions  points  one 
through  five,  scenario  combinations,  and  presentation  orders.  The 
analysis  indicates  a  significant  three  way  interaction  among  aiding/no- 
aiding,  A/NA  scenarios,  and  the  five  decision  points,  £  (4,  64)  =  2.83, 
2  <  .05.  Using  the  procedure  recommended  by  Keppel  (1973,  p.  293),  the 
three-way  interaction  was  further  analyzed,  by  breaking  it  down  into  two 
simple  two-way  interactions,  one  for  the  A  scenarios  and  the  other  for 
the  NA  scenarios.  The  simple  two-way  interaction  between  the 
aiding/no-aiding  factor  and  the  decision  point  factor  under  the  A 
scenarios,  shown  in  the  left-hand  panel  of  Figure  2,  is  statistically 
reliable,  £  (4,  64)  =  3.58,  _£  <  .025.  However,  the  interaction  between 
the  same  two  factors,  shown  in  the  right-hand  panel  of  Figure  2,  for  the 
NA  scenarios  is  just  short  of  statistical  significance  reliable  £  (4, 

64)  =  2.14,  .05  <£  <  .10). 

With  respect  to  the  absolute  level  of  confidence  placed  on  the  decision 
alternative  that  was  thought  to  be  correct  (i.e.,  the  highest  value  in 
the  confidence  vector),  the  overall  mean  level  was  64%  (62%  for  aided 
subjects  and  66%  for  unaided  subjects);  and  average  decision  confidence 
was  somewhat  higher  for  the  A  scenarios  (68%)  than  for  the  NA  scenarios 
(60%).  Given  the  corresponding  levels  of  percent  correct  decisions  (see 
Figure  1),  it  is  clear  that  the  degree  of  overstatement  of  confidence 
(i.e.,  difference  between  mean  confidence  and  percent  correct  decisions) 
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is  consistent  with  the  results  for  the  other  performance  measures.  That 
is,  there  was  generally  less  overstatement  of  confidence  in  the  two 
better  performance  conditions  (aided  group  in  A  scenarios,  and  unaided 
group  in  NA  scenarios)  than  in  the  two  poorer  performance  conditions 
(unaided  group  in  A  scenarios,  and  aided  group  in  NA  scenarios). 

Utl 1 ity  judgments.  The  utilities  (i.e.,  regret  values)  of  the  aided  and 
unaided  subjects  were  analyzed  to  determine  whether  any  differences  in 
decision  making  performance  between  the  groups  might  be  attributable  to 
differences  in  their  utility  functions.  The  aided  subjects'  utilities 
were  elicited  prior  to  the  decision  making  tasks,  while  the  unaided  sub¬ 
jects  utilities  were  elicited  at  the  end  of  the  experiment.  Mean  regret 
values  by  state-of-the-worl d  and  decision  options  for  aided  and  not  aid¬ 
ed  subjects  are  reported  in  Table  5.  A  comparison  between  the  utilities 
of  aided  versus  unaided  subjects  shows  that  they  do  not  differ  markedly; 
in  fact,  a  Mann-Whitney  jJ  test  does  not  indicate  that  the  utilities  are 
drawn  from  different  populations.  One  striking  aspect  of  these  data  is 
that  regret  values  are  much  1 arger  for  the  A  state-of-the-world  than  for 
the  NA  state-of-the-world;  that  is,  the  data  indicate  that  the  intelli¬ 
gence  officers  appear  to  be  much  more  concerned  about  being  attacked 
when  unprepared  than  being  overprepared  when  no  attack  will  occur.  In 
essence,  the  utilities  indicate  that  the  analysts  are  extremely  conser¬ 
vative  in  their  decision  responses  vis-a-vis  the  possibility  of  an  at¬ 
tack. 

Prior  probability  and  likelihood  ratio  estimates.  The  purpose  of  these 
analyses  is  to:  (a)  determine  whether  aided  analysts,  after  reading 
only  the  scenario  background  information,  are  biased  toward  either  an  A 
or  NA  state-of-the-world,  as  a  function  of  the  scenario  type  (A  or  NA); 
and,  (b)  investigate  the  impact  of  the  intelligence  summaries  on  the 
analysts'  estimates  of  the  likelihood  of  the  states-of-the-world.  The 


TABLE  5 


MEAN  REGRET  VALUES 


STATE-OF- 

DECISION 

GROUP 

THE-WORLD 

ALTERNATIVE 

AIDED 

UNAIDED 

ATTACK 

No  Alert 

-94.1 

-100.0 

Military 

Vigilance 

-59.6 

-74.6 

Simple 

Alert 

-33.0 

-35.3 

Rei nforced 

Alert 

-2.5 

-7.9 

NO  ATTACK 

No  Alert 

-5.8 

-3.5 

Miliary 

Vigilance 

-6.0 

-1.3 

Simple 

Alert 

-11.4 

-1.5 

Reinforced 

Alert 

-27.6 

-7.9 
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aided  subjects'  a  priori  probability  estimates  of  an  enemy  attack  for 
the  A  and  NA  scenarios,  elicited  after  the  subjects  had  read  the 
scenario  background  data,  are  very  similar.  The  estimates  are  P(A)  = 

.38  (SD  =  .17)  for  the  former  and  P(A)  =  .40  (SD  =  .17)  for  the  latter, 
and  the  difference  between  the  two  is  not  reliable;  these  values,  in¬ 
cidentally,  are  somewhat  lower  than  those  obtained  in  the  scenario  vali¬ 
dation  study  and  they  reflect  intentional,  subsequent  modifications  made 
to  the  scenario  background  material.  Across  the  intelligence  summaries, 
the  likelihood  ratio  estimates  increase  monotonical ly ;  this  result  is 
not  surprising  and  indicates  that  the  summaries  progressively  reflect 
the  predetermined  state-of-the-world.  In  raw  score  form,  the  likelihood 
ratios  averaged  across  scenarios  for  intelligence  summaries  one  through 
five  are  .90,  1.32,  2.32,  3.37,  and  7.14.  These  values  are  quite  simi¬ 
lar  to  the  average  likelihood  ratio  medians  computed  across  scenarios  in 
the  preliminary  validation  study,  namely,  1.14,  1.81,  2.57,  3.61,  7.60, 
respectively. 

Because  the  distribution  of  likelihood  ratio  data  are  negatively  skewed, 
the  data  were  subjected  to  a  logarithmic  transformati on  prior  to  per¬ 
forming  statistical  analyses.  The  data  were  analyzed  using  an  hierarch¬ 
ical  ANOVA  with  the  following  factors:  A  versus  NA,  decision  points  one 
through  five,  scenario  combinations,  and  presentation  orders.  The 
analysis  shows  that  the  likelihood  ratio  estimates  differ  signficantly 
across  the  intelligence  summaries,  £  (4,  32)  =  12.82,  _£  <  .001.  All 
other  factors  in  the  analysis  are  not  statistically  reliable. 

Decision  aid  versus  aided  subjects.  This  analysis  was  performed  because 
it  was  observed  that  in  many  cases  the  aided  subjects  chose  to  counter¬ 
mand  the  decision  aid.  At  issue  is  whether  the  subjects  performed 
better  or  worse  as  a  result  of  overriding  the  aid.  Overall,  the  aided 
subjects  accepted  the  aid's  recommendation  in  66.7%  of  the  cases.  The 
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decision  aid  recommended  an  average  of  51.6%  correct  decisions  while  the 
aided  subjects  attained  an  average  of  only  38.4%  correct  decisions. 

This  represents  a  34%  advantage  for  the  aid,  indicating  that  in  a  sub¬ 
stantial  number  of  cases  the  aided  subjects  would  have  performed  better 
had  they  followed  the  aid's  recommendation. 

A  hierarchical  four-way  ANOVA  was  conducted  comparing  the  mean  percent 
correct  decisions  attained  by  the  aided  analysts  versus  by  the  decision 
aid.  The  other  factors  in  the  ANOVA  were  A  versus  NA  scenarios, 
scenario  combinations,  and  presentation  orders.  The  difference  between 
the  decision  aid  and  the  aided  analysts  is  reliable,  £  (1,  8)  =  7.53, 
j)  <  .05.  Furthermore,  significantly  more  correct  decisions  are  attained 
under  the  A  scenarios  (83%  by  aid,  67%  by  analysts)  than  under  the  NA 
scenarios  (10%  by  aid,  20%  by  analysts),  £  (1,  8)  =  27.57,  <  .001. 

All  other  factors  in  the  ANOVA  are  not  statistically  reliable. 

To  suggest  possible  reasons  why  the  aided  subjects  chose  to  countermand 
the  decision  aid,  a  further  analysis  was  conducted  comparing  the  deci¬ 
sion  options  selected  by  the  aided  subjects  versus  those  recommended  by 
the  aid.  Table  6  displays  the  results;  entries  not  along  the  diagonal 
represent  recommendations  by  the  aid  that  were  countermanded.  Subjects 
followed  77%  and  70%  of  the  aid's  recommendations  in  the  A  and  NA 
scenario  versions,  respectively.  One  striking  aspect  of  the  data  re¬ 
ported  in  the  table  is  the  high  frequency  with  which  the  aid  recommended 
a  reinforced  alert  in  both  the  A  and  NA  versions.  The  data  also  show 
that,  in  the  A  versions,  in  7  out  of  48  cases  (15%)  where  the  aid  recom¬ 
mended  a  reinforced  alert,  subjects  preferred  the  less  extreme  "simple 
alert"  alternative.  Similarly,  in  the  NA  version,  in  9  of  15  cases 
(60%)  where  the  aid  recommended  a  "simple  alert"  subjects  preferred  the 
less  extreme  "military  vigilance"  alternative.  In  general,  aided  sub¬ 
jects  preferred  less  extreme  (risky)  decision  alternatives  relative  to 
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TABLE  6 


FREQUENCIES  OF  AIDED  RECOMMENDATION  BY  AIDED  SUBJECT  CHOICES 


DECISION  AID 

RECOMMENDATION 

SCENARIO 

TYPE 

SUBJECT 

CHOICE 

NO 

ALERT 

MILITARY 

VIGILANCE 

SIMPLE 

ALERT 

REINFORCED 

ALERT 

ATTACK 

No  Alert 

m 

0 

1 

0 

Military 

Vigilance 

i 

m 

3 

1 

Simple 

Alert 

0 

1 

m 

7 

Reinforced 

Alert 

0 

0 

0 

Ron* 

Total 

4 

1 

7 

48 

NO  ATTACK 

No  Alert 

El* 

1 

0 

0 

Mi  1 itary 
Vigilance 

2 

nn 

9 

1 

Simple 

Alert 

0 

0 

5 

Rei nforced 
Alert 

0 

0 

0 

1  21  | 

Total 

7 

11 

15 

27 

*  Frequency  of  correct  decisions. 
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the  decision  aid.  In  terms  of  correct  decisions,  it  should  be  noted 
that  the  data  in  Table  6  indicate  that  while  subjects  could  have  per¬ 
formed  better  by  following  the  aid,  there  is  only  one  case  where  a  sub¬ 
ject  performed  better  despite  the  aid,  that  is,  where  a  subject  counter¬ 
manded  the  aid  and,  as  a  result,  attained  a  correct  decision. 
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GENERAL  DISCUSSION  AND  CONCLUSIONS 


The  overall  pattern  of  results  provides  a  coherent  but  perplexing  pic¬ 
ture.  On  one  hand,  the  use  of  the  decision  aid  dramatically  improves 
decision  making  in  the  attack  (A)  scenarios,  especially  at  the  early  de¬ 
cision  points  where  the  amount  of  diagnostic  information  is  meager  and 
the  true  state-of-the-world  is  harder  to  discern.  It  should  be  em¬ 
phasized  that  this  represents  a  lower  bound  on  the  aided  subjects'  per¬ 
formance  given  that  they  could  have  performed  even  better  had  they  con¬ 
sistently  followed  the  aid's  recommendations.  On  the  other  hand,  the 
aided  subjects  accrue  no  advantage,  and  in  fact  perform  somewhat  more 
poorly,  compared  to  the  unaided  subjects  in  the  no-attack  (NA) 
scenarios.  In  agreement  with  Slovic  (1972),  these  findings  suggest  that 
the  quality  of  unaided  human  decision  making  and  the  potential  for  aid¬ 
ing  it  can  depend  upon  parameters  of  the  problem  situation. 

Several  post  hoc  explanations  may  be  advanced  to  account  for  the  differ¬ 
ences  in  performance  between  the  aided  and  the  unaided  subjects  in  one, 
but  not  the  other,  type  of  scenario.  As  one  possibility,  in  a  situation 
where  little  information  is  available,  the  aiding  process  may  operate 
more  effectively  when  the  information  has  a  high  rather  than  a  low  de¬ 
gree  of  diagnosticity.  Events  leading  to  an  attack  state-of-the-world 
are  general ly  more  diagnostic  than  events  pointing  to  a  no  attack 
state-of-the-world.  Information  indicative  of  enemy  intentions  to  at¬ 
tack  usually  involve  the  movement  of  troops,  military  equipment,  and  ma¬ 
teriel.  In  contrast,  an  enemy  intention  to  not  attack  is  usually 
characterized  by  the  absence  of  significant  belligerent  actions,  the 
maintenance  of  the  status  quo,  and  the  occurrence  of  diplomatic  negotia¬ 
tions.  Thus,  events  leading  to  an  attack  are  perceived  as  being  more 
salient  and  less  ambiguous  than  events  leading  to  no  attack;  and  this 
hypothesis  was  supported  by  the  data  obtained  in  the  scenario  evaluation 


study  ( see  Tabl e  2) . 


Along  the  same  lines,  performance  differences  between  unaided  and  aided 
analysts  in  the  A  scenarios  may  be  attributable  to  the  use  of  the  Baye¬ 
sian  revision  model  by  the  latter  group.  In  the  A  scenarios,  as  well  as 
in  the  NA  scenarios,  aided  subjects  did  appropriately  revise  their  as¬ 
sessments  of  the  probability  of  an  enemy  attack.  In  contrast,  the 
unaided  analysts  did  not  explicitly  state  probability  estimates;  and 
whatever  implicit  judgments  they  did  make  appeared  to  follow  the  pattern 
of  intuitive  conservative  inference  ( e . g . ,  DuChame,  1970).  In  any 
event,  since  the  unaided  subjects  did  not  provide  probability  state¬ 
ments,  the  experimental  design  did  not  allow  for  isolating  the  impact  of 
the  Bayesian  revision  model  on  decision  performance  within  the  context 
of  the  OPINT  package-- i .e. ,  it  cannot  be  determined  whether  an  effect  is 
due  to  the  availability  of  model-revised  probabilities  or  simply  to  the 
subjects'  assessment  of  conditional  probabil ity  judgments.  However, 
based  on  the  results  of  many  other  related  research  studies  (see  Beach, 
1975,  for  a  review),  it  is  reasonable  to  assume  that  the  Bayesian  revi¬ 
sion  model  had  a  beneficial  effect  on  performance,  at  least  for  the  A 
scenarios.  On  the  other  hand,  since  aided  analysts  effectively  used  the 
Bayesian  revision  model  in  the  NA  scenarios  as  well  (but  did  not  as  a 
result  make  better  decisions  than  unaided  analysts),  it  would  seem  that 
the  use  of  this  model  is  not  a  sufficient  factor  for  improving  decision 
performance.  Furthermore,  these  results  suggest  that  utility  considera¬ 
tions  can,  in  certain  situations,  be  much  more  important  in  influencing 
decision  behavior  than  subjective  probability  judgments. 

The  utility  judgments  or  regret  values  elicited  from  the  aided  and 
unaided  subjects  indicate  that  both  groups  of  analysts  are  conservative. 
That  is,  they  strongly  value  covering  themselves  against  an  impending 
attack,  the  case  which  obviously  carries  more  serious  military  conse- 
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quences;  this  result,  however,  might  be  a  consequence  of  the  artificial¬ 
ity  of  the  task  environment,  i.e.,  subjects  were  not  penalized  in  any 
way  for  inappropriate  decisions  nor  was  a  "cost"  incurred  when  a  deci¬ 
sion  recommendation  resulted  in  the  "taking  of  unnecessary  preparatory 
action."  As  part  of  the  aiding  process,  utilities  were  elicited  from, 
and  directly  considered  by,  the  aided  subjects.  However,  because  the 
unaided  subjects'  utilities  were  elicited  only  at  the  end  of  the  experi¬ 
ment  it  may  be  that  they  did  not  take  utility  judgments  into  acount  in 
the  decision  making  processs.  According  to  this  hypothesis,  aided  sub¬ 
jects  were  more  inclined  to  adopt  higher  levels  of  alert  and,  therefore, 
attain  more  correct  decisions  in  the  A  scenarios,  but  fewer  correct  de¬ 
cisions  in  the  NA  scenarios.  This  notion  is  very  much  consistent  with 
the  finding  of  Gettys,  May,  and  O'Bar  (1976),  that  the  use  of  a  decision 
aid  was  significantly  more  beneficial  to  those  individuals  who  demon¬ 
strated  a  greater  aversion  to  risk-taking  when  placed  in  a  high  risk  si¬ 
tuation.  Thus,  the  elicitation  procedure  alone  may  make  people  aware  of 
their  utilities  and  affect  performance  quite  independently  from  the  im¬ 
pact  of  their  seeing  the  results  of  the  SEU  aggregation  process. 

Familiarization  with  the  scenario  background  and  message  context  did  not 
affect  decision  making  performance.  That  is,  performance  with  respect 
to  all  dependent  measures  was  equivalent  for  the  familiar  Balkan 
scenario  and  unfamiliar  Shamba  scenario.  Although  one  could  hypothesize 
that  decision  aiding  might  have  a  more  pronounced  effect  in  an  unfami¬ 
liar  versus  a  familiar  context,  the  failure  to  find  a  familiarization 
effect  in  the  present  study  is  consistent  with  the  results  of  other  ex¬ 
periments,  such  as  those  investigating  the  role  of  information  familiar¬ 
ization  in  the  "risky-shift"  paradigm  (Pruitt,  1971).  To  the  extent 
that  familiarization  context  can  be  compared  with  easy/hard  problems,  it 
is  interesting  to  note  that  Siegel  and  Madden  (1980)  obtained  equivocal 
results  concerning  the  impact  of  the  latter  on  the  effectiveness  and 


51 


perceived  value  of  a  computer-based  decision  aid. 


An  important  finding  here  as  well  as  elsewhere  (e.g.,  Steeb,  Artof, 
Crooks,  and  Weltman,  1975)  is  that  aided  analysts  choose  to  countermand 
the  decision  aid  in  a  substantial  number  of  cases,  thereby  selecting 
inappropriate  decision  alternatives.  The  reasons  why  people  choose  to 
countermand  a  normative  decision  model  or  aid  are  not  clear  (e.g., 
Lichtenstein,  Slovic,  and  Zink,  1969).  In  describing  the  complexity  of 
the  problem,  Miller,  Morris,  Smallwood,  Lansford,  and  Gibbons  (1980) 
state: 


"In  the  traditional  paradigm,  the  decision-maker  must 
decide  to  accept  or  reject  the  model.  If  he  rejects 
the  model,  he  makes  the  intuitive  decision;  if  he 
accepts  the  model,  he  chooses  the  model  decision. 

There  are  at  least  two  problems  with  this.  First, 
there  are  no  explicit  criteria  for  determining  whether 
to  accept  or  reject  the  model.  The  deci sion-maker  is 
often  placed  in  the  position  of  having  to  judge  a 
model  whose  technical  details  he  doesn't  really  under¬ 
stand.  Second,  and  perhaps  more  important,  in  reject¬ 
ing  either  the  model  or  his  intuition,  the  decision¬ 
maker  is  throwing  away  potentially  useful  information.' 
(p.  37) 


When  a  decision  maker  is  unable  to  understand  the  decision  aiding  model, 
he  may  feel  threatened  and  this  feeling  can  be  enhanced  by  both  the  per¬ 
ceived  power  of  the  decision  analyst  and  the  decision  makers  apprehen¬ 
sion  that  his  decision  policies  may  be  captured  by  simple  models  (Lock, 
1980).  Thus,  a  decision  maker  may  choose  to  overrule  the  aid  in  order 
to  exercise  his  authority  and  compensate  for  his  own  feeling  of  in¬ 
feriority.  Other  reasons,  suggested  by  Spector,  Hayes,  and  Crain 
(1976),  for  why  people  may  not  follow  the  recommendation  of  decision 
aids  include  the  following:  the  aid  may  simply  be  misunderstood,  or 
training  may  be  inadequate;  the  aiding  algorithms  may  not  be  trusted  nor 
considered  adequate;  the  aid  may  lack  the  facility  to  be  adapted  to  per- 
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sonal  styles  of  problem  solving;  poor  performance  during  an  exercise  may 
magnify  and  reinforce  resistance  to  an  aid.  Whatever  the  case,  the  is¬ 
sue  of  user  acceptance  of  recommendations  from  decision  aids  is  of  crit¬ 
ical  importance  and  requires  further  investigation. 

The  lack  of  confidence  in  the  decision  aid  is  also  apparent  from  the 
results  of  the  analysis  of  confidence  validity  which  indicate  that  aided 
analysts  do  not  always  provide  more  appropriate  confidence  judgments  in 
their  decisions  than  unaided  analysts.  In  fact,  the  data  show  that  in 
the  NA  scenarios  aided  subjects  greatly  overstated  their  confidence  in 
decision  correctiveness.  It  is  therefore  essential  to  build  up  the 
user's  confidence  in  the  aiding  procedure.  One  potentially  effective 
means  for  increasing  user  confidence  might  be  to  have  aided  decision 
makers  develop  and  appreciate  a  history  of  success  when  applying  aiding 
to  difficult  problems.  Confidence  might  also  be  boosted  by  increasing 
the  user's  familiarity  in  the  aiding  process  and  in  the  theory  of  norma¬ 
tive  decision  making. 

Future  Research 


This  study  differed  from  many  other  evaluations  of  decision  aiding  in 
that  decisions  were  assessed  in  terms  of  their  outcomes-- i .e. ,  according 
to  a  resultant  state-of-the-world  being  diagnosed  as  part  of  the 
problem-solving  process.  Within  an  experimental  setting,  this  approach 
simulates  to  some  extent  the  way  people  usually  evaluate  the  correctness 
of  decisions  taken  in  the  "real  world."  Other  studies  have  typically 
evaluated  the  quality  of  decision  making  on  the  basis  of  the  degree  to 
which  decision  makers  maximize  SEU  or  are  internally  consistent  vis  a 
vis  their  judgments  and  decisions.  As  such,  this  study  provides  impor¬ 
tant  data  toward  helping  us  to  begin  to  understand  the  circumstances 
under  which  decision  aiding  can  be  most  supportive.  However,  the 
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results  are  limited,  in  part,  to  the  scenarios  used  in  the  study,  and 
Winkler  and  Murphy  (1973)  caution  that  the  simplicity  and  artificiality 
of  experimental  situations  cast  doubt  on  the  justification  for  general¬ 
izing  the  results  of  these  experiments  to  more  realistic  influencial  and 
decision  making  environments.  To  expand  the  investigation  of  the  value 
of  decision  aiding,  therefore,  the  experimental  approach  adopted  here 
might  also  be  used  to  compare  aided  and  unaided  decision  making  within 
other  "real-world"  situations,  such  as  military  exercises,  where  the 
resultant  state-of-the-world  is  ultimately  known  to  the  experimenter. 

An  important  issue  which  has  recently  been  addressed  in  the  literature 
concerns  the  effort  involved  in  the  use  of  decision  analysis  (e.g., 
Fischhoff,  Slovic,  and  Lichtenstein,  1979;  Johnson,  1979)  and  the  cost- 
effectiveness  of  this  approach  to  decision  making  (e.g.,  Fischhoff, 
Goitein,  and  Shapira,  1979).  In  the  present  experiment,  no  attempt  was 
made  to  control  for  time-on-task;  in  fact,  on  the  average,  aided 
analysts  spent  more  than  twice  as  much  time  as  unaided  analysts  (approx¬ 
imately  4  hours  versus  1  1/2  hours)  working  on  the  decision  problems. 
Thus,  the  effect  of  the  aid  was  confounded  with  problem  solution  time, 
including  exposure  to  task  materials  and  aiding  procedures.  Future  ex¬ 
periments  will  be  required  to  control  for  the  possibility  that  an  in¬ 
crease  in  "time  on  task"  may  account  for  aiding  effectiveness  simply  be¬ 
cause  of  a  deeper  reflection  into  the  decision  problem  (Keeney  and  Raif- 
f  a ,  1976).  For  example,  another  control  group  of  unaided  analysts  could 
be  used;  these  individuals  might  be  instructed  to  prepare  a  brief  writ¬ 
ten  account  of  the  rationale  underlying  each  decision  they  reach  (cf. 
Bateson,  1966),  with  the  amount  of  time  spent  per  decision  approximately 
equal  to  that  used  by  decision-aided  subjects.  If  the  extent  of  problem 
analysis  resulting  from  differential  "time  on  task"  is  sufficient  to  ex¬ 
plain  aiding  effectiveness,  the  group  in  this  second  control  condition 
should  perform  as  well  as  the  completely  aided  group. 
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From  the  results  of  this  study,  a  componential  analysis  of  decision  aid¬ 
ing  functions  suggests  that  either  the  aggregation  of  subjective  input 
or  the  structuring  of  task  environment  (through  elicitation),  or  both 
acting  together,  may  account  for  the  difference  in  performance  of  aided 
subjects.  Experimental  manipulations  are  therefore  required  to  empiri¬ 
cally  determine  the  separate  contributions  of  aggregation  and  structure. 
These  conditions  might  include  providing  the  decision  maker  with  task 
structure  by  requiring  him  to  interact  with  various  elicitation  pro¬ 
grams.  However,  neither  the  aggregated  results  nor  the  decision  recom¬ 
mendations  typically  afforded  by  the  aid  need  be  displayed.  If  struc¬ 
turing  alone  enhances  performance,  then  the  use  of  aggregation  in  a 
decision-theoretic  aiding  system  may  be  an  attractive,  but  purely  op¬ 
tional  ,  feature. 

The  analysis  of  decision  aiding  functions  may  be  extended  to  encompass 
the  issue  of  detailed  versus  global  elicitation.  The  elicitation  pro¬ 
cedures  for  the  OPINT  system  investigated  here  required  subjects  to  es¬ 
timate  conditional  event  probabilities  and,  in  addition,  to  assess  util¬ 
ities  separately  for  each  attribute  dimension.  A  less  time-consuming 
and  perhaps  more  efficient  strategy  is  to  simplify  elicitation  by  re¬ 
quiring  "global"  estimates  (e.g.,  unconditional  probabilities  and 
overall  utilities).  Hence,  other  experimental  mani pul ations  could  be 
implemented  to  determine  the  contribution  of  detailed  elicitation  to 
aiding  effectiveness.  For  example,  subjects  in  a  separate  treatment 
group  could  be  required  to  provide  global  subjective  estimates,  and  as  a 
result,  would  receive  decision  aiding  computed  on  the  basis  of  "simpli¬ 
fied"  input.  If  aiding  effectiveness  is  not  reduced,  the  use  of  de¬ 
tailed  elicitation  may  be  unwarranted. 
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APPENDIX  A 
BALKAN  SCENARIO 


Background  Information  for  the  Balkan  Scenario 


The  reality  of  deployed  NATO  and  Warsaw  Pact  forces  in  Europe  inexorably 
poses  the  threat  of  tension  and  crisis  escalating  to  war.  Both  the  United 
States  and  the  Soviet  Union  have  vital  national  security  interests  in 
Europe  that  are  dramatically  reflected  in  their  military  contributions  in 
the  two  opposing  alliances.  Combined  with  the  Military  forces  of  other 
alliance/pact  members,  the  European  theater  is  composed  of  large,  modern, 
and  potentially  destructive  forces  unparalleled  in  the  history  of  warfare. 

In  summary,  force  composition  can  be  assumed  to  be  approximately  as  follows: 


NORTHERN  AND  CENTRAL  EUROPE 

NATO 

WARSAW  PACT 

Combat  and  direct  support  troops  available 

625,000 

895,000 

Tanks 

7,000 

19,000 

Tactical  aircraft 

2,050 

4,025 

Nuclear  weapons 

7,000 

3,500 

Assume  it  is  late  winter,  1981.  A  number  of  recent  incidents,  beginning 
with  the  death  of  President  Tito  of  Yugoslavia  six  months  ago,  has  produced 
a  situation  that  may  seriously  affect  U.S.  interests  in  the  Balkans. 

Following  Tito's  demise,  Yugoslavia  has  been  ruled  by  a  collective  presi¬ 
dency  in  the  party.  Two  factions  within  the  presidency  have  hampered  the 
government  of  Yugoslavia.  One  faction  is  led  by  Stane  Dolanc  who  represents 
the  modern  technocrats  in  the  country.  Dolanc  is  a  spokesman  for  the 
economic  managers  and  intellectuals  and  favors  a  decentralized  confederate 
structure.  The  other  faction  is  headed  by  Branko  Mikulifc,  an  ideologi¬ 
cally  orthodox  hardliner,  and  is  supported  by  the  internal  security  establish¬ 
ment  and  the  army.  Mikulifc  advocates  a  firm  hand  against  dissent  and  favors 
central ization  in  Belgrade.  The  two  factions’  conflicting  views  of  the 
future  course  of  Yugoslavia  has  resulted  in  the  mutual  cancellation  of 
power  and  a  vacuum  of  leadership  has  developed.  During  the  deadlock,  the 
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BULGARIA 


members  of  an  illegal  pro-Soviet  party  have  begun  to  proselytize  openly. 
They  are  presumed  to  be  receiving  clandestine  support  from  the  Soviet 
Union  by  way  of  Bulgaria.  In  addition,  the  pro-Soviets  have  made  a  public 
appeal  for  aid  from  the  USSR. 

More  recently,  the  Soviet  Union,  in  an  attempt  to  capitalize  on  the 
apparent  weakness  of  the  Yugoslav  leadership,  has  sent  an  ultimatum  to 
Yugoslavia  demanding  the  establishment  of  several  naval  bases  in  that 
country  in  order  to  strengthen  their  position  in  the  Middle  East.  The 
ultimatum  in  conjunction  with  the  activities  of  the  pro- Soviet  communist 
party  has  resulted  in  public  anti-Soviet  demonstrations  by  the  Yugoslav 
population.  The  most  recent,  held  in  Belgrade  two  weeks  ago,  led  to 
severe  damage  to  the  Aeroflot  office.  The  CIA  also  reports  that  three 
GRU  (Soviet  military  intelligence)  officers  have  mysteriously  disappeared 
from  the  Soviet  naval  repair  facility  at  Montenegro  in  Yugoslavia.  Thus, 
Soviet  nations  in  Yugoslavia  are  in  some  danger. 

Earlier  this  year,  the  USSR  announced  that  some  Soviet  troops  would 
participate  in  the  Warsaw  Pact  military  exercises  that  would  be  held  in 
Bulgaria  this  year.  Although  Soviet  commanders  have  taken  part  in  the* 
Bulgarian  Warsaw  Pact  exercises,  large  numbers  of  Soviet  troops  have  never 
before  participated.  The  Rumanian  ambassador  in  Washington  in  a  private 
conversation  has  revealed  that  the  Kremlin  has  requested  permission  to 
move  ten  Soviet  divisions  across  Runanian  soil  to  the  staging  areas  in 
Bulgaria.  There  has  also  been  some  unrest  along  the  Bulgarian/Yugoslav 
border.  The  Bulgarian  communist  party  leader,  Todor  Zhivkov,  recently 
visited  a  number  of  towns  along  the  border  where  he  publicly  renewed 
dormant  claims  to  Yugoslav  Macedonia. 

As  the  intelligence  officer  attached  to  the  staff  of  CINCUSNAVREUR,  you 
have  been  ordered  by  your  commander  to  advise  him  about  the  miliary  and 
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political  events  in  Yugoslavia.  Currently,  the  situation  in  Yugoslavia  is 
ambiquous  and  complex.  There  is  always  a  possibility  that  the  Soviets 
will  take  direct  action.  Your  task  is  to  assess  all  incoming  information 
in  order  to  diagnose  the  developing  situation. 


BALKAN  INTELLIGENCE  SUMMARIES: 
ATTACK  VERSION 
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FIRST  INTELLIGENCE  SUMMARY:  Period  of  February  9  to  February  12 

A  reliable  British  source  reports  that  the  Warsaw  Pact 
air  commanders,  meeting  in  Budapest,  are  allegedly 
there  for  the  purpose  of  finalizing  this  year's  Warsaw 
Pact  exercise  rules. 

An  economic  mission  representing  two  NATO  coutries  is 
welcomed  in  Moscow. 

Strong  and  persistent  recent  Soviet  propaganda  states 
that  European  nations  should  quit  being  U.S.  puppets, 
that  they  should  not  depend  on  the  U.S.,  and  that  peace 
could  be  maintained  through  increased  European  cooper¬ 
ation  with  the  USSR  and  reduced  relations  with  the  U.S. 

SHAPE  intelligence  reports  serious  morale  problems  in 
the  Soviet  Navy.  The  major  sources  of  difficulty  are 
the  poor  living  conditions  of  Navy  personnel,  the  lack 
of  training  at  sea,  and  the  length  and  frequency  of 
breakdown  of  Soviet  Naval  equipment.  Soviet  ships 
spend  only  30%  of  their  time  at  sea;  the  rest  is  spent 
in  home  port. 

Bulgaria  alleges  two  border  violations  by  Yugoslav 
ai rcraft. 
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SECOND  INTELLIGENCE  SUMMARY:  Period  of  February  13  to  February  17 


The  CIA  reports  that  the  Warsaw  Pact  has  only  twelve 
days  of  POL  reserves  if  it  were  to  launch  an  attack 
against  Western  Europe  at  this  time.  It  is  estimated 
that  it  would  take  one  week  for  additional  supplies 
to  reach  the  Front  Edge  of  the  Battle  Area  (FEBA)  in 
Germany  from  the  USSR. 

PHOTINT  coverage  shows  that  the  amount  of  equipment 
removed  from  a  stationing  area  in  Hungary  has  been 
in  great  excess  of  that  regularly  observed  during 
Warsaw  Pact  maneuvers. 

The  Soviets  have  agreed  to  tentatively  schedule  the 
next  mutual  and  balanced  force  reduction  (MBFR)  dis¬ 
cussions  in  Vienna  for  October. 

German  intelligence  reports  that  a  greatly  increased 
number  of  Soviet  military  transport  specialists  are 
currently  arriving  in  Poland,  Hungary,  and  Czechoslo¬ 
vakia  than  are  usually  observed  during  Warsaw  Pact 
maneuvers. 

The  Chinese  attache  in  Moscow  reports  that  one  of  his 
sources  has  observed  that  some  Soviet  merchant  vessels 
in  the  Black  Sea  are  being  fitted  with  surface-to-sur¬ 
face  and  surface-to-air  missiles. 
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THIRD  INTELLIGENCE  SUMMARY:  Period  of  February  18  to  February  21 


The  Warsaw  Pact  military  leaders  assemble  for  a 
meeting  in  Moscow.  Pravda  states  that  the  meeting 
is  in  connection  with  the  Warsaw  Pact  exercise  plan. 
Such  high-level  review  has  not  been  observed  for 
previous  exercises. 

The  French  ambassador  to  Russia  is  called  to  the 
Kremlin  and  made  aware  of  the  Soviets'  concern  for 
mil itary-related  activities  which  they  have 
observed  in  France  and  have  interpreted  as 
increased  military  readiness  actions. 

U.S.  intelligence  reports  a  75%  build-up  of  Russian 
submarines  in  both  the  Atlantic  and  Pacific. 

PHOTINT  shows  that  ten  Soviet  divisions  are  moving 
to  railheads  near  the  Rumanian  border. 

Rumania  signs  a  friendship  agreement  with  China, 
covering  mutual  aid  and  increased  economic  coopera¬ 
tion  between  the  two  countries. 


FOURTH  INTELLIGENCE  SUMMARY:  Period  of  February  22  to  February  26 


Very  reliable  HUMINT  sources  in  Belgrade  and 
Montenegro  report  that  Soviet  dependents  are 
leaving  Yugoslavia. 

Two  Soviet  motorized  rifle  divisions  are  reported 
to  be  crossing  Rumania  en  route  to  the  exercise  area 
in  Bulgaria. 

A  COMINT  source  confirms  earlier  reports  of  the 
requisitioning  of  civilian  trucks  for  military  use 
by  the  Soviets  in  Bulgaria. 

A  PHOTINT  report  shows  that  Soviet  airborne  troops 
are  out  of  garrison  in  the  Western  USSR. 

The  Soviets  warn  the  U.S.  that  the  Sixth  Fleet  is 
operating  dangerously  close  in  the  Mediterranean 
and  is  interfering  with  Soviet  fleet  operations. 


FIFTH  INTELLIGENCE  SUMMARY:  Period  of  February  27  to  March  3 


Signed  articles  in  Kommunist  argue  strongly  that  the 
USSR  cannot  afford  a  guns-and-butter  economy  and  must 
provide  more  consumer  goods. 

PHOTINT  shows  that  Soviet  forces  in  Hungary  are  being 
redeployed  and  bolstered  by  four  Soviet  divisions 
from  the  Western  military  districts  of  the  USSR. 


PHOTINT  shows  that  the  Yugoslavs  are  moving  troops  from 
the  Italian  and  Greek  borders  to  face  Warsaw  Pact  troops 
on  the  Bulgarian  and  Hungarian  borders. 

Multiple  sources  indicate  a  world-wide  alert  of  Soviet 
forces. 

The  Yugoslav  government  calls  for  a  special  meeting  of 
the  United  Nations  General  Assembly  to  discuss  the 
Soviet  threat. 
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FIRST  INTELLIGENCE  SUMMARY:  Period  of  February  9  to  February  12 


The  Soviets  are  pushing  for  a  summit  meeting  this  year 
to  finalize  the  signing  of  a  SALT  II  pact  and  to  out¬ 
line  the  next  steps  in  detente. 

Naval  intelligence  reports  that  the  Soviets  are 
becoming  much  more  aggressive  in  shadowing  U.S. 
submarines  leaving  home  port.  Fourty  percent  more 
U.S.  submarines  are  currently  being  shadowed  than  are 
usually  shadowed. 

The  Kremlin  has  invited  eight  U.S.  Senators,  who  are 
undecided  on  how  to  vote  on  the  SALT  accords,  to  visit 
Moscow  next  month.  If  the  Senators  accept,  it  will 
provide  the  Kremlin  an  opportunity  to  exert  influence 
toward  the  ratification  of  the  treaty. 

SIGINT  analysis  shows  that  the  Warsaw  Pact  air  defense 
system  has  gone  on  maximum  alert.  This  may  be  related 
to  the  first  live  play  in  preparation  for  the  Warsaw 
Pact  exercises  in  Bulgaria  and  Hungary. 

The  Soviets  are  taking  a  conciliatory  strategy  towards 
the  Eurocommunists.  A  recent  Pravda  editorial  states 
that  communist  parties  attaining  power  in  European 
countries  can  mutually  coexist  with  the  world  wide 
communist  movement  if  they  keep  national  sovereignty 
and  internationalism  in  proper  balance. 
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ECOND  INTELLIGENCE  SUMMARY:  Period  of  February  13  to  February  17 

The  Soviet  ambassador  in  Vienna  has  made  a  strong 
diplomatic  move  vis-a-vis  the  Austrian  government 
to  remind  it  of  its  neutral  status,  especially 
towards  events  occurring  in  Yugoslavia. 

The  USSR  has  informed  the  Japanese  that  the  Soviets 
are  interested  in  renewing  talks  concerning  a  peace 
treaty  related  to  the  northern  territories  issue. 

The  northern  NATO  nations  are  sternly  warned  by  the 
USSR  to  keep  their  naval  units  out  of  the  Warsaw  Pact 
territorial  waters  of  the  Eastern  Baltic. 

Yugoslavia  proposes  to  hold  informal  discussions 
with  Bulgaria  concerning  the  Yugoslav  Macedonia  issue. 

Tourists  in  Moscow  have  noticed  that  government 
operated  stores  are  carrying  a  greater  variety  of 
consumer  goods.  The  Soviet  Union's  cut  in  defense 
spending,  beginning  18  months  ago,  is  now  being  trans¬ 
lated  into  more  products  of  better  quality  for  the 


consumer. 


THIRD  INTELLIGENCE  SUMMARY:  Period  of  February  18  to  February  21 

The  Rumanian  embassy  in  Washington  has  revealed  that 
the  Yugoslavs  have  requested  that  the  Rumanian 
ambassador  in  Moscow  play  the  role  of  intermediary 
between  the  Soviets  and  Yugoslavs  concerning  recent 
events  in  Yugoslavia. 

The  Soviet  news  agency  TASS  reports  that  the  Soviets 
will  resume  the  Strategic  Arms  Limitation  Treaty  (SALT) 
negotiations  in  order  to  overcome  U.S.  Senate  objections. 

Extensive  snowstorms  in  the  western  USSR  have  tied  up 
surface  transportation. 

Polish  Catholics  are  reported  to  be  pressing  the 
regime  for  greater  personal  freedom  in  that  country. 
Reputable  sources  report  meetings  between  Polish 
communist  party  leaders  and  Catholic  clergy  on  these 
issues. 

The  Warsaw  Pact  defense  ministers  assemble  for  a 
meeting  in  Moscow.  Pravda  states  that  the  meeting 
is  in  connection  with  the  Warsaw  Pact  exercise  plan. 

Such  high-level  review  has  not  been  reported  for  most 
Warsaw  Pact  exercises. 
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FOURTH  INTELLIGENCE  SUMMARY :  Period  of  February  22  to  February  26 


PHOTINT  shows  that  four  Soviet  divisions  are 
deploying  in  the  exercise  assembly  areas  in  Bulgaria. 
This  is  a  somewhat  lower  level  of  Soviet  participation 
in  the  exercise  than  was  anticiapted  earlier. 

A  Soviet  Foreign  Ministry  statement  broadcast  by  TASS 
has  expressed  concern  about  the  Yugoslav  situation 
but  has  dissociated  itself  from  calls  for  aid  from  the 
pro-Soviet  faction  in  Yugoslavia. 

The  Chinese  are  reported  to  be  arming  troops  on  the 
USSR/China  border  with  a  new  surface-to-surface  missile. 

Yugoslavia  proposes  to  allow  the  Soviets  to  expand 
their  naval  repair  facility  in  Montenegro  but  refuses 
to  allow  the  Soviets  to  establish  naval  bases  in 
Yugoslavia. 

North  Korean  President  Kim  II  Sung  met  with  Chinese 
leaders  in  Peking.  In  a  speech  delivered  before 
officials  of  the  Chinese  communist  party,  he  accused 
the  USSR  of  "hegemon ism"  in  Asia. 
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FIFTH  INTELLIGENCE  SUMMARY:  Period  of  February  27  to  March  3 

Naval  intelligence  reports  that  the  number  of  Soviet 
ships  in  the  Mediterranean  has  dropped  to  an  all-time 
low. 

It  has  been  announced  in  Sophia  that  Todor  Zhivkov 
(the  Communist  Party  Chief  in  Bulgaria)  has  been 
removed  because  of  nationalist  excesses. 

Yugoslavia  has  sent  a  top  level  delegation  to  Moscow 
to  conciliate  differences. 

The  Soviets  announce  that  the  Warsaw  Pact  exercises 
in  Bulgaria  have  been  shortened  due  to  poor  weather. 
Runanian  sources  report  that  one  Soviet  regiment  has 
already  returned  to  the  USSR. 

Soviet  ambassadors  in  key  Western  capitals  have 
requested  audiences  with  heads  of  state  regarding  the 
Yugoslav  crisis. 
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APPENDIX  B 
SHAMBA  SCENARIO 


Background  Information  for  Shamba  Scenario 


You  represent  a  powerful  nation  called  the  UNION  OF  NORTH  HEMISPHERIC 
STATES  (UNHS).  You  are  opposed  by  a  Rebel  Movement  in  the  country  of 
Shamba.  The  existence  of  this  movement  threatens  the  freedom  of  choice 
of  the  people  of  Shamba.  It  is,  in  effect,  part  of  a  world-wide  movement 
designed  to  destroy  the  freedom  we  have  learned  to  appreciate.  This  move¬ 
ment  is  supported  by  a  powerful  group  of  nations  called  the  Socialist 
Alliance  (SA). 

There  are  six  parties  involved  in  one  way  or  another  in  the  Shamba  con¬ 
flict.  They  are: 

The  Shamba  Military  Government.  Over  the  last  five  years,  there  have  been 
a  number  of  military  coups  d'etat  in  Shamba.  All  Shamba  administrations, 
however,  have  given  at  least  tacit  support  to  the  cause  of  ousting  the 
Rebels  from  Shamba.  The  present  government  has  been  in  power  for  three 
and  one- half  months  and  is  preparing  a  plan  for  economic  development  of 
certain  regions  of  the  country.  These  plans  are  hampered,  however,  by  the 
cost  of  maintaining  a  large  army  to  cope  with  the  Rebel  Movement. 

The  Rebel  Movement  (Rebels).  At  the  present  time,  the  Rebels  are  operating 
out  of  Marandi  and  Monque  Island,  where  they  have  been  in  exile  for  the 
past  five  years.  The  diplomatic  and  military  leaders  of  the  Rebel  Movement 
were  recently  replaced  by  new  men.  These  new  men  were  apparently  carefully 
selected  and  trained  by  the  Socialist  Alliance  for  their  current  positions 
the  Rebel  Movement. 

The  Socialist  Alliance  (SA).  The  Socialist  Alliance  is  a  major  world  power 
much  like  the  UNHS.  It  has  a  totalitarian  form  of  government,  and  its  aim 
is  to  foster  its  form  of  government  around  the  world.  In  the  past,  the  s 
Socialist  Alliance  has  not  supported  the  Rebel  cause  with  direct  military 


aid.  The  Socialist  Alliance  dominates  a  military  alliance  called  the 
Social ist  Axis. 


The  Country  of  Marandi .  Officially,  the  government  of  Marandi  is  neutral 
in  the  current  situation.  In  fact,  however,  they  have  tolerated  the 
presence  of  troop  training  operations  by  the  Rebels,  have  provided  a  haven 
for  fleeing  troops,  and  have  allowed  the  Rebels  to  maintain  headquarters  in 
Marandi.  At  this  time,  the  government  of  Marandi  maintains  full  diplomatic 
relations  with  the  military  government  of  Shamba.  But,  in  its  attempt  to 
industrialize,  Marandi  has  become  increasingly  dependent  on  economic  aid 
from  the  Socialist  Alliance. 

Mandero  City.  Mandero  is  an  independent  and  neutral  city-state  on  the 
coast  of  Shamba.  It  has  served  well  as  an  area  for  recreation,  as  well 
as  for  contacts  with  the  enemy.  The  UNHS,  as  well  as  the  Rebel  Movement 
has  guaranteed  the  independence  of  Mandero. 

The  UNHS.  Five  years  ago,  the  embattled  military  government  of  Shamba 
called  for  help  outside  the  country.  Your  nation,  the  Union  of  North 
Hemispheric  States  (UNHS),  agreed  to  aid  the  government  in  combatting 
the  rebellion.  In  return,  the  UNHS  asked  that  it  be  permitted  to 
establish  a  military  (naval  and  air)  base  at  Komsa.  The  Military 
National  Government  of  Shamba  agreed.  Since  that  time,  the  UNHS  has 
been  consulted  by  the  government  of  Shamba  concerning  most  of  the  im¬ 
portant  decisions  of  internal  affairs  in  Shamba  and  all  major  foreign 
policy  decisions. 

The  situation  in  which  you  find  yourself  at  the  beginning  of  the 
scenario  is  as  follows  (see  map  in  Figure  A-l): 

Shamba  has  been  governed  by  a  quasi-military  dictatorship  for  about 
twelve  years.  The  government,  formed  after  a  military  coup  d'etat,  has 
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become  unpopular  among  certain  segments  of  the  population.  Rebel  activ¬ 
ity  has  been  mounting.  Some  five  years  ago,  the  Rebel  Movement  was  in 
virtual  control  of  the  country,  with  the  exception  of  the  capital  city 
of  Savin.  The  Rebels  established  a  rival  government,  the  so-called 
"Free  Government  of  the  Nation  of  Shamba,"  which  considered  itself  to 
be  the  legal  government  of  the  population  of  Shamba.  At  that  time,  a 
number  of  foreign  nations,  among  them  Marandi,  recognized  the  Rebel 
Government  as  the  legitimate  governing  body  of  all  Shamba. 

With  the  help  of  UNHS  arms  and  some  troops,  the  Rebels  were  driven  back 
into  the  mountains,  onto  Monque  Island  where  they  have  established  a  pro¬ 
visional  capital  of  Shamba,  and  into  the  territory  of  Marandi.  Prior 
to  last  year,  only  small  gains  had  been  made  by  the  Rebels  since  the 
defeat  of  their  rebellion  five  years  ago.  One  year  ago,  the  rebellion 
again  became  quite  open,  with  the  Rebels'  military  strength  and  nego¬ 
tiating  position  improving,  due  to  increased  assistance  from  other 
nations,  especially  the  Socialist  Alliance  (SA)  by  way  of  Marandi. 

There  is  strong  enmity  between  Shamba  and  Marandi.  Over  one  hundred 
years  ago,  the  King  of  Shamba  invaded  Marandi  and  captured  McKosam  and 
the  area  south  of  the  city  to  the  present  Shamba/Marandi  border. 

Marandi  still  claims  the  territory  but  has  not  had  sufficient  military 
strength  to  recapture  it.  The  current  military  dictatorship  in  Marandi 
has  recently  revived  this  old  territorial  issue  and  as  a  result  has 
heightened  nationalist  fevor  in  Marandi.  The  Rebel  cause  in  Shamba  is 
supported  by  natives  in  the  McKosam  region  who  trace  their  loyalties 
and  ancestry  to  Marandi. 

Recently,  the  Rebels  were  responsible  for  the  complete  destruction  of 
a  village  near  Batu  where  over  a  hundred  women  and  children  were 
massacred.  This  was  followed  by  the  assasination  of  an  important  Rebel 
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leader  in  Marandi  City.  The  assasi nation  is  thought  to  have  been  executed 
by  a  commando  group  of  the  Shamba  Military  Government.  The  Rebels  have 
retaliated  by  escalating  propaganda  against  the  Shamba  Military  Government 
and  have  publicly  requested  the  assistance  of  Marandi  and  of  the  Socialist 
Alliance. 

Lately,  the  Shamba  Military  Government,  with  the  UNHS's  support,  has 
pursued  the  Rebels  into  Marandi.  In  reprisal,  Marandi  has  lodged  a  formal 
complaint  with  the  League  of  United  Nations,  signed  a  mutual  defense  pact 
with  the  Socialist  Alliance,  and  has  openly  expressed  its  support  for 
the  Rebel  Movement  in  Shamba. 

Over  the  past  five  years,  the  UNHS  has  steadily  increased  its  military 
commitments  to  Shamba  to  cope  with  the  worsening  situation.  As  the 
Naval  Intelligence  Officer  attached  to  the  UNHS  High  Command  in  Shamba, 
you  support  the  top  level  representatives.  In  order  to  support  UNHS 
objectives  in  Shamba,  your  task  is  to  assess  the  enemy's  intentions 
through  the  analysis  of  intelligence  reports.  You  should  be  prepared 
to  make  recommendations  to  your  superior  by  taking  both  military  and 
economic  factors  into  consideration.  Currently,  the  situation  in  the 
Shambanese  Peninsula  is  ambiguous  and  complex.  There  is  always  a  possi¬ 
bility  that  the  Rebels  will  launch  an  offensive  with  the  support  of 
their  allies.  You  are  asked,  therefore,  to  monitor  all  current  infor¬ 
mation  in  order  to  diagnose  the  developing  situation. 
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SHAMBA  INTELLIGENCE  SUMMARIES: 
ATTACK  VERSION 
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HtcCEDlNG  PAGE  BLANK 


FIRST  INTELLIGENCE  SUMMARY:  Period  of  May  2  to  May  6 

A  reliable  UNHS  ally  reports  that  the  rebel  leader¬ 
ship  has  met  recently  with  high-level  Socialist 
Alliance  military  advisers  for  the  purpose  of 
planning  Rebel  negotiation  and  military  strategy. 

An  economic  mission  representing  two  UNHS  allies 
has  been  welcomed  in  Marandi. 

Recent  Rebel  propaganda  states  that  Shamba  should 
quit  being  a  UNHS  puppet,  that  they  should  not 
depend  on  the  UNHS,  and  that  peace  could  be  ob¬ 
tained  by  supporting  the  current  Free  Government 
of  the  Nation  of  Shamba  against  Shamba  Military 
Government  and  the  UNHS. 

UNHS  intelligence  reports  serious  morale  problems 
in  the  rebel  camps  located  in  Marandi.  The  major 
sources  of  difficulty  are  insufficient  food,  poor 
medical  facilities,  and  a  shortage  of  medical 
suppl ies. 

Marandi  alleges  two  border  violations  by  Shambanese 
border  guards. 
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PRECEDING  PAGE  BUkNK-WOT  Flii'iED 


SECOND  INTELLIGENCE  SUMMARY:  Period  of  May  7  to  May  10 


The  UNHS  Central  Intelligence  Department  reports  that 
the  Rebels  have  insufficient  food  supplies  to  carry 
out  a  major  offensive  in  Shamba  at  this  time. 

Several  very  reliable  HUMINT  sources  reports  that  the 
Rebels  in  Marandi  camps  are  receiving  an  increased 
amount  of  weapons  and  ammunition.  These  include  small 
arms  and  anti-tank  weapons. 

The  Rebels  have  agreed  to  schedule  the  next  round  of 
negotiations  with  the  Shamba  Military  Government  in 
Mandero  City  during  July. 

UNHS  military  inte1 ' igence  reports  that  there  are  a 
greater  number  of  Socialist  Alliance  advisers  in 
Marandi  City  than  usual. 

✓ 

The  Mandero  attache  in  Marandi  reports  that  one  of 
his  sources  has  observed  a  small  shipment  of  arms 
being  unloaded. from  a  Socialist  Alliance  merchant 
vessel  in  Marandi  City  Harbor. 


THIRD  INTELLIGENCE  SUMMARY :  Period  of  May  11  to  May  15 


A  reliable  source  reports  that  a  general  meeting  of 
the  rebel  leadership  was  recently  held  i r.  a  rebel 
camp  along  the  Ondulu  River.  Such  a  high-level  meet¬ 
ing  has  never  before  been  reported. 

The  Mandero  City  ambassador  to  the  Socialist  Alliance 
is  made  aware  of  the  Socialist  Alliance's  concern  for 
military-related  activities  which  they  have  observed  in 
Mandero  City  and  have  interpreted  as  increased  military 
readiness  actions. 

UNHS  intelligence  reports  a  50%  build-up  of  Socialist 
Alliance  Navy  ships  in.  the  Western  waters  off  Shamba 
and  around  Monque  Island. 

PHOTINT  shows  that  there  is  far  less  activity  in  one 
Rebel  camp  in  Marandi  along  the  Ondulu  River  than 
usual.  It  is  thought  that  the  Rebels  from  this  camp 
have  infiltrated  back  into  Shamba. 

Mandero  City  signs  a  friendship  agreement  with  Lota, 
a  large  independent  country  with  little  military 
power,  covering  mutual  aid  and  increased  economic 
cooperation  between  the  two  countries. 
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FOURTH  INTELLIGENCE  SUMMARY:  Period  of  May  16  to  May  19 


A  very  reliable  HUMINT  source  in  Savin  and  McKosam 
reports  that  dependents  of  Marandi  officials  living 
in  Shamba  are  leaving  the  country. 

A  regiment  of  Marandi  troops  is  reported  to  be 
moving  from  the  south  of  Marandi  to  a  position  near 
the  Marandi/Shamba  border. 

A  COMINT  source  confirms  earlier  reports  of  the 
requisitioning  of  civilian  trucks  for  military  use 
by  the  Marandi  Army  in  Marandi  City. 

A  PHOTINT  report  shows  that  another  Marandi  regiment 
is  moving  toward  the  Marandi/Shamba  border. 

The  Socialist  Alliance  warns  that  the  UNHS  fleet 
in  the  waters  west  of  Shamba  is  interfering  with 
Socialist  Alliance  fleet  exercises. 
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FIFTH  INTELLIGENCE  SUMMARY:  Period  of  May  20  to  May  24 


Signed  articles  in  The  Socialist  argue  strongly  that 
the  Socialist  Alliance  cannot  afford  a  guns-and-butter 
economy  and  as  a  result  must  provide  more  consumer 
goods  and  reduce  military  aid  to  foreign  countries. 

UNHS  military  intelligence  reports  an  increased  number 
of  engagements  with  rebel  forces  along  the  Marandi  City/ 
McKosam  railroad  and  in  the  vicinity  of  Komsa. 

UNHS  military  intelligence  reports  that  two  Marandi 
and  one  Socialist  Alliance  advisers  were  captured 
in  an  engagement  with  the  Rebels  near  Batu. 

Multiple  sources  indicate  that  the  Socialist  Alliance 
Fleet  off  the  western  shore  of  Shamba  has  been  placed 
on  alert. 

The  Shamba  government  calls  for  a  special  meeting  of 
the  League  of  United  Nations  General  Assembly  to 
discuss  the  Marandi  and  Socialist  Alliance  threat 
to  Shamba. 
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SHAMBA  INTELLIGENCE  SUMMARIES: 
NO  ATTACK  VERSION 
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FIRST  INTELLIGENCE  SUMMARY:  Period  of  May  2  to  May  6 


The  Socialist  Alliance  is  pushing  for  a  summit  with 
the  UNHS  this  year  to  sign  a  treaty  which  will  mutually 
limit  the  number  and  types  of  weapon  systems  sold  to 
developing  countries. 

Naval  intelligence  reports  that  the  Socialist  Alliance 
is  becoming  more  aggressive  in  shadowing  UNHS  aircraft 
carriers  leaving  home  port.  Sixty-percent  more  aircraft 
carriers  are  being  shadowed  than  usual. 

The  President  of  Marandi  has  invited  four  UNHS  legisla¬ 
tors  to  visit  Marandi  City  next  month.  The  visit  of 
UNHS  legislators  to  Marandi  City  will  provide  the 
President  of  Marandi  with  an  opportunity  to  renew  his 
request  for  "most  favored  status"  in  commercial  relations 
with  the  UNHS. 

ELINT  analysis  shows  that  a  much  greater  number  of  small 
aircraft  landings  than  usual  have  occurred  along  the 
Marandi/Shamba  border.  The  aircraft  are  suspected  to 
be  transporting  arms  and  munitions  to  the  Rebels. 

The  Socialist  Alliance  is  taking  a  conciliatory  strategy 
towards  independent  socialist  countries.  A  recent  edi¬ 
torial  in  The  Socialist  states  that  socialist  parties 
attaining  power  in  non-socialist  countries  can  mutually 
coexist  with  the  world  wide  socialist  movement  if  they 
keep  national  sovereignty  and  internationalism  in  proper 
balance. 
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SECOND  INTELLIGENCE  SUMMARY:  Period  of  May  7  to  May  10 

The  Socialist  Alliance  in  Mandero  City  has  made 
strong  diplomatic  moves  vis-a-vis  the  Mandero  City 
Government  to  remind  it  of  its  neutral  status.  The 
Socialist  Alliance  does  not  want  Mandero  City  to  close 
its  border  to  fleeing  Rebels. 

The  Socialist  Alliance  has  informed  Giad,  a  neigh¬ 
boring  country,  that  it  is  interested  in  renewing 
talks  concerning  the  ownership  of  a  number  of  islands 
under  dispute  between  the  two. 

Shamba  and  the  UNHS  have  been  sternly  warned  by 
Marandi  to  keep  their  naval  units  out  of  Marandi's 
territorial  waters. 

The  Shamba  Military  Government  proposes  to  hold 
informal  discussions  with  Marandi  concerning  areas 
under  dispute  along  the  Shamba/Marandi  border. 

Tourists  in  the  Socialist  Alliance  have  noticed 
that  government  operated  stores  are  carrying  a 
greater  variety  of  consumer  goods.  The  Socialist 
Alliance's  cut  in  defense  spending,  beginnging  18 
months  ago,  is  now  being  translated  into  more 
products  of  better  quality  for  the  consumer. 
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THIRD  INTELLIGENCE  SUMMARY:  Period  of  May  11  to  May  15 


The  Mandero  City  embassy  in  the  capital  of  the  UNHS 
has  revealed  that  the  Rebels  have  requested  that 
the  Mandero  City  government  play  the  role  of  inter¬ 
mediary  between  the  Rebels  and  the  Shamba  Military 
Government  concerning  the  possibility  of  peace  talks. 

The  Socialist  Alliance  news  agency  MUN  reports 
that  the  Socialists  will  resune  negotiations  for 
a  treaty  to  mutually  limit  arms  to  developing 
countries  in  order  to  overcome  political  objections 
raised  in  the  UNHS. 

Extensive  rains  in  the  Shamba/Marandi  peninsula 
have  caused  many  rivers  to  become  impassable  and 
have  destroyed  many  bridges  along  the  Marandi , 

Ondulu,  Batu,  and  Savinari  rivers. 

Religious  groups  in  Marandi  are  reported  to  be 
pressing  the  regime  for  greater  personal  freedom 
in  that  country.  Reputable  sources  report  meetings 
between  the  party  leaders  and  religious  leaders  on 
these  issues. 

A  reliable  source  reports  that  a  general  meeting 
of  the  Rebel  military  leadership  was  recently  held 
in  a  Rebel  camp  along  the  Ondulu  River.  Such  a 
high-level  meeting  has  never  before  been  reported. 
Important  Socialist  Alliance  military  advisors  are 
also  thought  to  have  participated. 


FOURTH  INTELLIGENCE  SUMMARY:  Period  of  May  16  to  May  19 


A  HUMINT  source  reports  that  the  number  of 
Socialist  Alliance  advisors  is  increasing 
in  Marandi. 

A  Socialist  Alliance  Foreign  Ministry  statement 
broadcast  by  the  foreign  news  agency  MUN  has 
expressed  concern  about  the  Rebel  situation  in 
Shamba  but  has  dissociated  itself  from  calls  for 
aid  on  the  part  of  the  Rebels. 

Giad,  a  large  non-aligned  neighbor  to  the  Socialist 
Alliance,  is  reported  to  be  arming  troops  on  the 
Giad/Socialist  Alliance  border  with  a  new  surface- 
to-surface  missile. 

Marandi  proposes  to  allow  the  Socialist  Alliance 
to  expand  their  naval  repair  facility  in  Marandi 
City  Harbor  but  refuses  to  allow  the  Alliance  to 
establish  a  naval  base  there. 

A 

The  Mandero  City  president  accuses  the  UNHS  and 
the  Socialist  Alliance  of  practicing  "hegemonism" 
in  the  Shamba/Marandi  peninsula. 
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FIFTH  INTELLIGENCE  SUWARY:  Period  of  May  20  to  May  24 


Naval  intelligence  reports  that  the  number  of 
Socialist  Alliance  ships  in  the  waters  surrounding 
Shamba  has  dropped  to  an  all-time  low. 

The  Socialist  Alliance's  Minister  for  Foreign 
Affairs  publicly  denounced  Marandi  for  national¬ 
istic  excesses  in  its  dispute  with  Shamba. 

The  Shamba  Military  Government  has  sent  a  top-level 
delegation  to  conciliate  differences  with  Marandi 
concerning  recent  border  incidents. 

Marandi  announces  that  it  will  take  a  harder 
position  against  Shamba  Rebels  taking  refuge  in 
Marandi.- 

Socialist  Alliance  ambassadors  in  key  UNHS  ally 
capitals  have  requested  audiences  with  heads-of- 
state  regarding  the  Socialist  Alliance's  position 
on  the  Shamba/Marandi  dispute. 
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COMPUTATIONAL  FORMULA  FOR  RANK  PROBABILITY  SCORE 


The  formula  for  computing  the  rank  probability  score  (RPS),  according  to 
Epstein  (1969),  is  as  follows: 


„  ,  K-l  i  0  K  5  ,  K 

Si  =  f  “  2  TFTT  ^  K  l  p/  +  {  l  Pn)  ]-FT  l  M'Jl  Pi 
j  ^  c  In  lj  .=1  n.j  n  n=i+1  n  \  i  i=1  i 


where 


S.  =  ranked  probability  score  of  the  subjective  probability  or  confidence 
J 

vector  (P^,  P3»  •••  p^)  w^en  alternative  j  is  correct,  and 

K  =  number  of  alternatives 
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